[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:38:38 dist-test-slave-zv8z krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:38:38 dist-test-slave-zv8z krb5kdc[399](info): set up 2 sockets
Sep 27 02:38:38 dist-test-slave-zv8z krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 02:38:38.994321 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/krb5kdc/test-user.keytab.
Sep 27 02:38:39 dist-test-slave-zv8z krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:38:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:38:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:38:39.121608 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35611
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35611
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:38:39.227473 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:38:39.227737 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:38:39.227793 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:38:39.231778 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:38:39.231842 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:38:39.231869 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:38:39.231889 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:38:39.231909 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:38:39.237128 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:38:39.238121 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:38:39.239136 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:38:39.247926 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:38:39.247946 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:38:39.247946 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:38:39.248003 412 server_base.cc:1034] running on GCE node
I20260927 02:38:39.248783 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:38:39.249855 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:38:39.251029 412 hybrid_clock.cc:648] HybridClock initialized: now 1790476719250962 us; error 76 us; skew 500 ppm
Sep 27 02:38:39 dist-test-slave-zv8z krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:38:39.253695 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:38:39.254611 412 webserver.cc:460] Webserver started at http://127.0.98.62:45263/ using document root <none> and password file <none>
I20260927 02:38:39.255111 412 fs_manager.cc:359] Metadata directory not provided
I20260927 02:38:39.255167 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:38:39.255352 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:38:39.257294 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/master-0/data/instance:
uuid: "1a13038e58eb47df8ce3722e9c4a36f7"
format_stamp: "Formatted at 2026-09-27 02:38:39 on dist-test-slave-zv8z"
server_key: "242d803421604d921def679134eb484c"
server_key_iv: "f283fd7636ba654fe38116c0700146fe"
server_key_version: "encryptionkey@0"
I20260927 02:38:39.257788 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/master-0/wal/instance:
uuid: "1a13038e58eb47df8ce3722e9c4a36f7"
format_stamp: "Formatted at 2026-09-27 02:38:39 on dist-test-slave-zv8z"
server_key: "242d803421604d921def679134eb484c"
server_key_iv: "f283fd7636ba654fe38116c0700146fe"
server_key_version: "encryptionkey@0"
I20260927 02:38:39.261318 412 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 02:38:39.263762 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:39.264947 412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:38:39.265054 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "1a13038e58eb47df8ce3722e9c4a36f7"
format_stamp: "Formatted at 2026-09-27 02:38:39 on dist-test-slave-zv8z"
server_key: "242d803421604d921def679134eb484c"
server_key_iv: "f283fd7636ba654fe38116c0700146fe"
server_key_version: "encryptionkey@0"
I20260927 02:38:39.265148 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:38:39.282375 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:38:39.285539 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:38:39.285709 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:38:39.292470 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35611
I20260927 02:38:39.292560 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35611 every 8 connection(s)
I20260927 02:38:39.293449 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:38:39.296247 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:38:39.297580 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 412
I20260927 02:38:39.297746 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/master-0/wal/instance
I20260927 02:38:39.299384 392 external_mini_cluster.cc:1428] Setting key 0e07aa1e0b4a67b837c54dbb1ec16266
I20260927 02:38:39.303264 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a13038e58eb47df8ce3722e9c4a36f7: Bootstrap starting.
I20260927 02:38:39.306082 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a13038e58eb47df8ce3722e9c4a36f7: Neither blocks nor log segments found. Creating new log.
I20260927 02:38:39.307013 480 log.cc:824] T 00000000000000000000000000000000 P 1a13038e58eb47df8ce3722e9c4a36f7: Log is configured to *not* fsync() on all Append() calls
I20260927 02:38:39.308908 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a13038e58eb47df8ce3722e9c4a36f7: No bootstrap required, opened a new log
I20260927 02:38:39.312258 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a13038e58eb47df8ce3722e9c4a36f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a13038e58eb47df8ce3722e9c4a36f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35611 } }
I20260927 02:38:39.312470 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a13038e58eb47df8ce3722e9c4a36f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:38:39.312542 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a13038e58eb47df8ce3722e9c4a36f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a13038e58eb47df8ce3722e9c4a36f7, State: Initialized, Role: FOLLOWER
I20260927 02:38:39.312943 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a13038e58eb47df8ce3722e9c4a36f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a13038e58eb47df8ce3722e9c4a36f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35611 } }
I20260927 02:38:39.313074 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a13038e58eb47df8ce3722e9c4a36f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:38:39.313136 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a13038e58eb47df8ce3722e9c4a36f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:38:39.313226 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a13038e58eb47df8ce3722e9c4a36f7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:38:39.314054 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a13038e58eb47df8ce3722e9c4a36f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a13038e58eb47df8ce3722e9c4a36f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35611 } }
I20260927 02:38:39.314397 480 leader_election.cc:302] T 00000000000000000000000000000000 P 1a13038e58eb47df8ce3722e9c4a36f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a13038e58eb47df8ce3722e9c4a36f7; no voters:
I20260927 02:38:39.314672 480 leader_election.cc:288] T 00000000000000000000000000000000 P 1a13038e58eb47df8ce3722e9c4a36f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:38:39.314750 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a13038e58eb47df8ce3722e9c4a36f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:38:39.314935 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a13038e58eb47df8ce3722e9c4a36f7 [term 1 LEADER]: Becoming Leader. State: Replica: 1a13038e58eb47df8ce3722e9c4a36f7, State: Running, Role: LEADER
I20260927 02:38:39.315282 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a13038e58eb47df8ce3722e9c4a36f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a13038e58eb47df8ce3722e9c4a36f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35611 } }
I20260927 02:38:39.315930 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a13038e58eb47df8ce3722e9c4a36f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:38:39.316870 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a13038e58eb47df8ce3722e9c4a36f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a13038e58eb47df8ce3722e9c4a36f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a13038e58eb47df8ce3722e9c4a36f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35611 } } }
I20260927 02:38:39.316998 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a13038e58eb47df8ce3722e9c4a36f7 [sys.catalog]: This master's current role is: LEADER
I20260927 02:38:39.317332 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a13038e58eb47df8ce3722e9c4a36f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a13038e58eb47df8ce3722e9c4a36f7. Latest consensus state: current_term: 1 leader_uuid: "1a13038e58eb47df8ce3722e9c4a36f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a13038e58eb47df8ce3722e9c4a36f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35611 } } }
I20260927 02:38:39.317476 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a13038e58eb47df8ce3722e9c4a36f7 [sys.catalog]: This master's current role is: LEADER
Sep 27 02:38:39 dist-test-slave-zv8z krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:38:39.317780 494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:38:39.321267 494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:38:39.323513 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:39.303056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42984 (local address 127.0.98.62:35611)
0927 02:38:39.303532 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:39.303552 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:39.304579 (+ 1027us) server_negotiation.cc:407] Connection header received
0927 02:38:39.308440 (+ 3861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:39.308456 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:39.308805 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:39.309103 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:39.311233 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:39.312181 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:39.313409 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:39.313851 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:39.313991 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:39.318640 (+ 4649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:39.318646 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:39.318662 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:39.318692 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:39.320299 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:39.320784 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:39.320788 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:39.320792 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:39.320899 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:39.321213 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:39.321216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:39.321217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:39.321613 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:39.321795 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:39.322373 (+ 578us) server_negotiation.cc:299] Negotiation successful
0927 02:38:39.322632 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":139,"threads_started":1}
I20260927 02:38:39.326941 494 catalog_manager.cc:1347] Generated new cluster ID: 5ba3c1ff756f4b26b2b676485302d269
I20260927 02:38:39.327001 494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:38:39.339665 494 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:38:39.340654 494 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:38:39.345922 494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a13038e58eb47df8ce3722e9c4a36f7: Generated new TSK 0
I20260927 02:38:39.346728 494 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:38:39.387108 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35611
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:38:39.496008 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:38:39.496302 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:38:39.496363 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:38:39.500456 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:38:39.500523 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:38:39.500613 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:38:39.505978 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35611
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:38:39.507123 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:38:39.508154 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:38:39.517637 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:38:39.517647 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:38:39.517648 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:38:39.517918 506 server_base.cc:1034] running on GCE node
I20260927 02:38:39.518458 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:38:39.519057 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:38:39.520236 506 hybrid_clock.cc:648] HybridClock initialized: now 1790476719520217 us; error 35 us; skew 500 ppm
Sep 27 02:38:39 dist-test-slave-zv8z krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:38:39.522987 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:38:39.523875 506 webserver.cc:460] Webserver started at http://127.0.98.1:37095/ using document root <none> and password file <none>
I20260927 02:38:39.524394 506 fs_manager.cc:359] Metadata directory not provided
I20260927 02:38:39.524457 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:38:39.524664 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:38:39.526644 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/ts-0/data/instance:
uuid: "16b4024a2b904c2183c1deaed34d039f"
format_stamp: "Formatted at 2026-09-27 02:38:39 on dist-test-slave-zv8z"
server_key: "a24b39a0bd8b193b7f08eba01ac4f6cd"
server_key_iv: "f8daa4e90a1caf1b5a0403a8d6fefa1e"
server_key_version: "encryptionkey@0"
I20260927 02:38:39.527160 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/ts-0/wal/instance:
uuid: "16b4024a2b904c2183c1deaed34d039f"
format_stamp: "Formatted at 2026-09-27 02:38:39 on dist-test-slave-zv8z"
server_key: "a24b39a0bd8b193b7f08eba01ac4f6cd"
server_key_iv: "f8daa4e90a1caf1b5a0403a8d6fefa1e"
server_key_version: "encryptionkey@0"
I20260927 02:38:39.530632 506 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 02:38:39.532961 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:39.534067 506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:38:39.534161 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/ts-0/wal
uuid: "16b4024a2b904c2183c1deaed34d039f"
format_stamp: "Formatted at 2026-09-27 02:38:39 on dist-test-slave-zv8z"
server_key: "a24b39a0bd8b193b7f08eba01ac4f6cd"
server_key_iv: "f8daa4e90a1caf1b5a0403a8d6fefa1e"
server_key_version: "encryptionkey@0"
I20260927 02:38:39.534296 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:38:39.548102 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:38:39.550962 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:38:39.551127 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:38:39.551695 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:38:39.552516 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:38:39.552560 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:39.552599 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:38:39.552626 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:39.560979 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46141
I20260927 02:38:39.561021 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46141 every 8 connection(s)
I20260927 02:38:39.561846 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:38:39.563565 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 506
I20260927 02:38:39.563657 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:38:39.563879 392 external_mini_cluster.cc:1428] Setting key 8861138a97a133115522c18a30eedce7
Sep 27 02:38:39 dist-test-slave-zv8z krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 02:38:39.575376 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:39.563641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50895 (local address 127.0.98.62:35611)
0927 02:38:39.563794 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:39.563798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:39.564654 (+ 856us) server_negotiation.cc:407] Connection header received
0927 02:38:39.566139 (+ 1485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:39.566143 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:39.566194 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:39.566287 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:39.567413 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:39.567955 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:39.569279 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:39.569500 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:39.569586 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:39.572041 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:39.572044 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:39.572046 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:39.572068 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:39.573337 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:39.573832 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:39.573835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:39.573836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:39.573904 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:39.574200 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:39.574203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:39.574205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:39.574433 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:39.574524 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:39.575131 (+ 607us) server_negotiation.cc:299] Negotiation successful
0927 02:38:39.575237 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 02:38:39.576332 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:39.564060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35611 (local address 127.0.98.1:50895)
0927 02:38:39.564510 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:39.564560 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:39.565920 (+ 1360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:39.566466 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:39.566472 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:39.566822 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:38:39.567276 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:39.567288 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:39.568072 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:39.568076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:39.569157 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:39.569166 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:39.569629 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:39.569633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:39.569800 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:39.570435 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:38:39.570450 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:38:39.571880 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:38:39.573441 (+ 1561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:39.573446 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:39.573461 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:39.573737 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:39.573983 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:39.573986 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:39.573987 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:39.574117 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:39.574533 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:38:39.574539 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:38:39.574859 (+ 320us) client_negotiation.cc:769] Sending connection context
0927 02:38:39.575062 (+ 203us) client_negotiation.cc:240] Negotiation successful
0927 02:38:39.575401 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":117,"threads_started":1}
I20260927 02:38:39.577713 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35611
I20260927 02:38:39.577957 635 heartbeater.cc:464] Registering TS with master...
I20260927 02:38:39.578469 635 heartbeater.cc:511] Master 127.0.98.62:35611 requested a full tablet report, sending...
I20260927 02:38:39.580245 444 ts_manager.cc:194] Registered new tserver with Master: 16b4024a2b904c2183c1deaed34d039f (127.0.98.1:46141)
I20260927 02:38:39.581820 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50895
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:38:39.590124 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35611
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:38:39.698580 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:38:39.698863 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:38:39.698921 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:38:39.702908 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:38:39.702970 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:38:39.703052 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:38:39.708246 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35611
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:38:39.709232 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:38:39.710211 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:38:39.719620 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:38:39.719637 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:38:39.719659 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:38:39.719854 640 server_base.cc:1034] running on GCE node
I20260927 02:38:39.720232 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:38:39.720824 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:38:39.721982 640 hybrid_clock.cc:648] HybridClock initialized: now 1790476719721963 us; error 31 us; skew 500 ppm
Sep 27 02:38:39 dist-test-slave-zv8z krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:38:39.724493 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:38:39.725299 640 webserver.cc:460] Webserver started at http://127.0.98.2:37737/ using document root <none> and password file <none>
I20260927 02:38:39.725800 640 fs_manager.cc:359] Metadata directory not provided
I20260927 02:38:39.725855 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:38:39.726020 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:38:39.727926 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/ts-1/data/instance:
uuid: "b02d1ca76aa04efaa2750b642ec55f6a"
format_stamp: "Formatted at 2026-09-27 02:38:39 on dist-test-slave-zv8z"
server_key: "b7bf99a9857f815a3f765a6b661e9bb6"
server_key_iv: "d6b20408e5788eccd449d460faeeeb4c"
server_key_version: "encryptionkey@0"
I20260927 02:38:39.728405 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b02d1ca76aa04efaa2750b642ec55f6a"
format_stamp: "Formatted at 2026-09-27 02:38:39 on dist-test-slave-zv8z"
server_key: "b7bf99a9857f815a3f765a6b661e9bb6"
server_key_iv: "d6b20408e5788eccd449d460faeeeb4c"
server_key_version: "encryptionkey@0"
I20260927 02:38:39.731771 640 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:38:39.734092 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:39.735342 640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 02:38:39.735462 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/ts-1/wal
uuid: "b02d1ca76aa04efaa2750b642ec55f6a"
format_stamp: "Formatted at 2026-09-27 02:38:39 on dist-test-slave-zv8z"
server_key: "b7bf99a9857f815a3f765a6b661e9bb6"
server_key_iv: "d6b20408e5788eccd449d460faeeeb4c"
server_key_version: "encryptionkey@0"
I20260927 02:38:39.735577 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:38:39.753448 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:38:39.756464 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:38:39.756626 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:38:39.757144 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:38:39.757877 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:38:39.757922 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:39.757966 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:38:39.757987 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:39.766788 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43111
I20260927 02:38:39.766822 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43111 every 8 connection(s)
I20260927 02:38:39.767616 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:38:39.776232 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 640
I20260927 02:38:39.776340 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:38:39.776628 392 external_mini_cluster.cc:1428] Setting key 9d95b383af55ab70155c70414c34b19c
Sep 27 02:38:39 dist-test-slave-zv8z krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:38:39.781045 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:39.769322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35173 (local address 127.0.98.62:35611)
0927 02:38:39.769464 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:39.769468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:39.770360 (+ 892us) server_negotiation.cc:407] Connection header received
0927 02:38:39.771621 (+ 1261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:39.771624 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:39.771669 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:39.771748 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:39.772845 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:39.773359 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:39.774538 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:39.774796 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:39.774886 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:39.777274 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:39.777278 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:39.777281 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:39.777307 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:39.778872 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:39.779413 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:39.779417 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:39.779419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:39.779550 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:39.779926 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:39.779929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:39.779931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:39.780154 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:39.780229 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:39.780825 (+ 596us) server_negotiation.cc:299] Negotiation successful
0927 02:38:39.780927 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 02:38:39.782133 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:39.769738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35611 (local address 127.0.98.2:35173)
0927 02:38:39.770204 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:39.770243 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:39.771424 (+ 1181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:39.771884 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:39.771891 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:39.772248 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:38:39.772708 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:39.772720 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:39.773482 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:39.773486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:39.774401 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:39.774410 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:39.774937 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:39.774940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:39.775097 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:39.775702 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:38:39.775717 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:38:39.777132 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:38:39.779031 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:39.779035 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:39.779048 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:39.779300 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:39.779682 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:39.779684 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:39.779686 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:39.779827 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:39.780247 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:38:39.780253 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:38:39.780563 (+ 310us) client_negotiation.cc:769] Sending connection context
0927 02:38:39.780784 (+ 221us) client_negotiation.cc:240] Negotiation successful
0927 02:38:39.781122 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":105,"threads_started":1}
I20260927 02:38:39.783329 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35611
I20260927 02:38:39.783542 769 heartbeater.cc:464] Registering TS with master...
I20260927 02:38:39.783993 769 heartbeater.cc:511] Master 127.0.98.62:35611 requested a full tablet report, sending...
I20260927 02:38:39.785044 444 ts_manager.cc:194] Registered new tserver with Master: b02d1ca76aa04efaa2750b642ec55f6a (127.0.98.2:43111)
I20260927 02:38:39.785776 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35173
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:38:39.803031 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35611
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:38:39.909576 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:38:39.909828 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:38:39.909870 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:38:39.913740 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:38:39.913795 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:38:39.913851 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:38:39.919157 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35611
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:38:39.920104 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:38:39.921034 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:38:39.930146 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:38:39.930284 774 server_base.cc:1034] running on GCE node
W20260927 02:38:39.930156 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:38:39.930492 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:38:39.930871 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:38:39.931491 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:38:39.932672 774 hybrid_clock.cc:648] HybridClock initialized: now 1790476719932644 us; error 35 us; skew 500 ppm
Sep 27 02:38:39 dist-test-slave-zv8z krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:38:39.935387 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:38:39.936208 774 webserver.cc:460] Webserver started at http://127.0.98.3:34591/ using document root <none> and password file <none>
I20260927 02:38:39.936718 774 fs_manager.cc:359] Metadata directory not provided
I20260927 02:38:39.936774 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:38:39.936929 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:38:39.939141 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/ts-2/data/instance:
uuid: "98423703ded54e5c8f62f39d2ab8afff"
format_stamp: "Formatted at 2026-09-27 02:38:39 on dist-test-slave-zv8z"
server_key: "26fb5dce8a5e005e9b49afa230e319fb"
server_key_iv: "91ae5af576fe294813181d6826fd83c6"
server_key_version: "encryptionkey@0"
I20260927 02:38:39.939635 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/ts-2/wal/instance:
uuid: "98423703ded54e5c8f62f39d2ab8afff"
format_stamp: "Formatted at 2026-09-27 02:38:39 on dist-test-slave-zv8z"
server_key: "26fb5dce8a5e005e9b49afa230e319fb"
server_key_iv: "91ae5af576fe294813181d6826fd83c6"
server_key_version: "encryptionkey@0"
I20260927 02:38:39.943225 774 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:38:39.945596 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:39.946897 774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:38:39.946995 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/ts-2/wal
uuid: "98423703ded54e5c8f62f39d2ab8afff"
format_stamp: "Formatted at 2026-09-27 02:38:39 on dist-test-slave-zv8z"
server_key: "26fb5dce8a5e005e9b49afa230e319fb"
server_key_iv: "91ae5af576fe294813181d6826fd83c6"
server_key_version: "encryptionkey@0"
I20260927 02:38:39.947093 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:38:39.957880 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:38:39.960954 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:38:39.961131 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:38:39.961694 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:38:39.962513 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:38:39.962560 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:39.962602 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:38:39.962677 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:39.971807 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41263
I20260927 02:38:39.971827 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41263 every 8 connection(s)
I20260927 02:38:39.972739 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:38:39.978283 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 774
I20260927 02:38:39.978390 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790476718023131-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:38:39.978654 392 external_mini_cluster.cc:1428] Setting key 0cd177e4a0742a74b16385881ac933d1
Sep 27 02:38:39 dist-test-slave-zv8z krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:38:39.985993 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:39.974481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59365 (local address 127.0.98.62:35611)
0927 02:38:39.974679 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:39.974683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:39.975505 (+ 822us) server_negotiation.cc:407] Connection header received
0927 02:38:39.977045 (+ 1540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:39.977048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:39.977097 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:39.977176 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:39.978345 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:39.979083 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:39.980437 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:39.980691 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:39.980770 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:39.983060 (+ 2290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:39.983063 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:39.983065 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:39.983089 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:39.984214 (+ 1125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:39.984649 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:39.984651 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:39.984652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:39.984716 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:39.984970 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:39.984973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:39.984974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:39.985151 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:39.985222 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:39.985755 (+ 533us) server_negotiation.cc:299] Negotiation successful
0927 02:38:39.985852 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260927 02:38:39.986812 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:39.974909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35611 (local address 127.0.98.3:59365)
0927 02:38:39.975341 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:39.975383 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:39.976750 (+ 1367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:39.977344 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:39.977350 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:39.977703 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:38:39.978192 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:39.978205 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:39.979232 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:39.979235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:39.980335 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:39.980343 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:39.980826 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:39.980829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:39.980984 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:39.981475 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:38:39.981498 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:38:39.982921 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:38:39.984307 (+ 1386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:39.984311 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:39.984322 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:39.984573 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:39.984773 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:39.984775 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:39.984776 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:39.984898 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:39.985259 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:38:39.985265 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:38:39.985501 (+ 236us) client_negotiation.cc:769] Sending connection context
0927 02:38:39.985657 (+ 156us) client_negotiation.cc:240] Negotiation successful
0927 02:38:39.985903 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":106,"threads_started":1}
I20260927 02:38:39.987861 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35611
I20260927 02:38:39.988093 903 heartbeater.cc:464] Registering TS with master...
I20260927 02:38:39.988591 903 heartbeater.cc:511] Master 127.0.98.62:35611 requested a full tablet report, sending...
I20260927 02:38:39.989526 444 ts_manager.cc:194] Registered new tserver with Master: 98423703ded54e5c8f62f39d2ab8afff (127.0.98.3:41263)
I20260927 02:38:39.990068 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59365
I20260927 02:38:39.992560 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 02:38:39 dist-test-slave-zv8z krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476719, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 02:38:40 dist-test-slave-zv8z krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476719, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:38:40.018093 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:40.008845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51488 (local address 127.0.98.1:46141)
0927 02:38:40.009170 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:40.009180 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:40.009202 (+ 22us) server_negotiation.cc:407] Connection header received
0927 02:38:40.009250 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:40.009254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:40.009443 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:40.009561 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:40.010199 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:40.011072 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:40.011946 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:40.012323 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:40.012410 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:40.014890 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:40.014899 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:40.014913 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:40.014938 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:40.016465 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:40.016871 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:40.016876 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:40.016877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:40.016971 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:40.017165 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:40.017167 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:40.017168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:40.017428 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:40.017597 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:40.017769 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 02:38:40.017932 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":131,"threads_started":1}
W20260927 02:38:40.019096 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51488
Sep 27 02:38:40 dist-test-slave-zv8z krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 02:38:40 dist-test-slave-zv8z krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:38:40.043886 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:40.035608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51492 (local address 127.0.98.1:46141)
0927 02:38:40.035737 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:40.035742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:40.035898 (+ 156us) server_negotiation.cc:407] Connection header received
0927 02:38:40.036084 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:40.036086 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:40.036133 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:40.036198 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:40.036903 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:40.037442 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:40.038348 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:40.038577 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:40.038664 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:40.041052 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:40.041055 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:40.041057 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:40.041085 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:40.042398 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:40.042846 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:40.042849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:40.042850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:40.042926 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:40.043130 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:40.043133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:40.043135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:40.043292 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:40.043388 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:40.043637 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 02:38:40.043731 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 02:38:40.044847 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 506
I20260927 02:38:40.050074 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 640
I20260927 02:38:40.056579 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 774
I20260927 02:38:40.061187 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 412
2026-09-27T02:38:40Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2030 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:38:40 dist-test-slave-zv8z krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:38:40 dist-test-slave-zv8z krb5kdc[917](info): set up 2 sockets
Sep 27 02:38:40 dist-test-slave-zv8z krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260927 02:38:41.017731 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/krb5kdc/test-user.keytab.
Sep 27 02:38:41 dist-test-slave-zv8z krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:38:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:38:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:38:41.131364 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39585
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39585
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:38:41.236474 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:38:41.236742 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:38:41.236796 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:38:41.240741 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:38:41.240804 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:38:41.240830 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:38:41.240850 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:38:41.240870 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:38:41.246029 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:38:41.247040 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:38:41.247996 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:38:41.255825 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:38:41.255868 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:38:41.255887 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:38:41.256035 930 server_base.cc:1034] running on GCE node
I20260927 02:38:41.256595 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:38:41.257614 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:38:41.258767 930 hybrid_clock.cc:648] HybridClock initialized: now 1790476721258746 us; error 30 us; skew 500 ppm
Sep 27 02:38:41 dist-test-slave-zv8z krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476721, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:38:41.261389 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:38:41.262152 930 webserver.cc:460] Webserver started at http://127.0.98.62:34921/ using document root <none> and password file <none>
I20260927 02:38:41.262677 930 fs_manager.cc:359] Metadata directory not provided
I20260927 02:38:41.262748 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:38:41.262907 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:38:41.264758 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/master-0/data/instance:
uuid: "751503cdf87e4920942c05712c4b083a"
format_stamp: "Formatted at 2026-09-27 02:38:41 on dist-test-slave-zv8z"
server_key: "4501a07b54c3108c442d9b3720add0a7"
server_key_iv: "e7183bb17f2769a3bcb526ad0b6b6c43"
server_key_version: "encryptionkey@0"
I20260927 02:38:41.265231 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/master-0/wal/instance:
uuid: "751503cdf87e4920942c05712c4b083a"
format_stamp: "Formatted at 2026-09-27 02:38:41 on dist-test-slave-zv8z"
server_key: "4501a07b54c3108c442d9b3720add0a7"
server_key_iv: "e7183bb17f2769a3bcb526ad0b6b6c43"
server_key_version: "encryptionkey@0"
I20260927 02:38:41.268620 930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 02:38:41.270841 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:41.272056 930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 02:38:41.272185 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "751503cdf87e4920942c05712c4b083a"
format_stamp: "Formatted at 2026-09-27 02:38:41 on dist-test-slave-zv8z"
server_key: "4501a07b54c3108c442d9b3720add0a7"
server_key_iv: "e7183bb17f2769a3bcb526ad0b6b6c43"
server_key_version: "encryptionkey@0"
I20260927 02:38:41.272287 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:38:41.283622 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:38:41.286608 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:38:41.286775 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:38:41.293197 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39585
I20260927 02:38:41.293265 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39585 every 8 connection(s)
I20260927 02:38:41.294107 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:38:41.296741 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:38:41.297003 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 930
I20260927 02:38:41.297107 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/master-0/wal/instance
I20260927 02:38:41.297360 392 external_mini_cluster.cc:1428] Setting key 6f2b8a517ee93aa66e07b11d0a87fa8d
I20260927 02:38:41.303649 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 751503cdf87e4920942c05712c4b083a: Bootstrap starting.
I20260927 02:38:41.305946 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 751503cdf87e4920942c05712c4b083a: Neither blocks nor log segments found. Creating new log.
Sep 27 02:38:41 dist-test-slave-zv8z krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:38:41.306706 998 log.cc:824] T 00000000000000000000000000000000 P 751503cdf87e4920942c05712c4b083a: Log is configured to *not* fsync() on all Append() calls
I20260927 02:38:41.308534 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 751503cdf87e4920942c05712c4b083a: No bootstrap required, opened a new log
I20260927 02:38:41.311424 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:41.298731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51868 (local address 127.0.98.62:39585)
0927 02:38:41.299134 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:41.299153 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:41.299193 (+ 40us) server_negotiation.cc:407] Connection header received
0927 02:38:41.299934 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:41.299957 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:41.300367 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:41.300723 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:41.301394 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:41.302898 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:41.303768 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:41.304404 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:41.304548 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:41.307074 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:41.307080 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:41.307093 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:41.307120 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:41.308696 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:41.309150 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:41.309155 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:41.309160 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:41.309297 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:41.309567 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:41.309569 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:41.309571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:41.309955 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:41.310094 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:41.310408 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 02:38:41.310685 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":107,"threads_started":1}
I20260927 02:38:41.312373 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 751503cdf87e4920942c05712c4b083a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751503cdf87e4920942c05712c4b083a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39585 } }
I20260927 02:38:41.312636 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 751503cdf87e4920942c05712c4b083a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:38:41.312690 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 751503cdf87e4920942c05712c4b083a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 751503cdf87e4920942c05712c4b083a, State: Initialized, Role: FOLLOWER
I20260927 02:38:41.313135 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 751503cdf87e4920942c05712c4b083a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751503cdf87e4920942c05712c4b083a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39585 } }
I20260927 02:38:41.313292 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 751503cdf87e4920942c05712c4b083a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:38:41.313352 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 751503cdf87e4920942c05712c4b083a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:38:41.313447 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 751503cdf87e4920942c05712c4b083a [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:38:41.314365 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 751503cdf87e4920942c05712c4b083a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751503cdf87e4920942c05712c4b083a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39585 } }
I20260927 02:38:41.314652 998 leader_election.cc:302] T 00000000000000000000000000000000 P 751503cdf87e4920942c05712c4b083a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 751503cdf87e4920942c05712c4b083a; no voters:
I20260927 02:38:41.314904 998 leader_election.cc:288] T 00000000000000000000000000000000 P 751503cdf87e4920942c05712c4b083a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:38:41.315016 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 751503cdf87e4920942c05712c4b083a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:38:41.315222 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 751503cdf87e4920942c05712c4b083a [term 1 LEADER]: Becoming Leader. State: Replica: 751503cdf87e4920942c05712c4b083a, State: Running, Role: LEADER
I20260927 02:38:41.315526 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 751503cdf87e4920942c05712c4b083a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751503cdf87e4920942c05712c4b083a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39585 } }
I20260927 02:38:41.316110 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 751503cdf87e4920942c05712c4b083a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:38:41.317154 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 751503cdf87e4920942c05712c4b083a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 751503cdf87e4920942c05712c4b083a. Latest consensus state: current_term: 1 leader_uuid: "751503cdf87e4920942c05712c4b083a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751503cdf87e4920942c05712c4b083a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39585 } } }
I20260927 02:38:41.317158 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 751503cdf87e4920942c05712c4b083a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "751503cdf87e4920942c05712c4b083a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "751503cdf87e4920942c05712c4b083a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39585 } } }
I20260927 02:38:41.317305 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 751503cdf87e4920942c05712c4b083a [sys.catalog]: This master's current role is: LEADER
I20260927 02:38:41.317305 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 751503cdf87e4920942c05712c4b083a [sys.catalog]: This master's current role is: LEADER
I20260927 02:38:41.317690 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:38:41.320616 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:38:41.325815 1012 catalog_manager.cc:1347] Generated new cluster ID: 08b0af5b1af949c9a7c204d3fffd2648
I20260927 02:38:41.325881 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:38:41.332487 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:38:41.333391 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:38:41.339769 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 751503cdf87e4920942c05712c4b083a: Generated new TSK 0
I20260927 02:38:41.340476 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:38:41.366081 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39585
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:38:41.474854 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:38:41.475096 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:38:41.475138 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:38:41.479171 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:38:41.479224 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:38:41.479283 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:38:41.484418 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39585
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:38:41.485446 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:38:41.486517 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:38:41.495950 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:38:41.495927 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:38:41.496059 1024 server_base.cc:1034] running on GCE node
W20260927 02:38:41.495927 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:38:41.496590 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:38:41.497205 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:38:41.498328 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790476721498292 us; error 45 us; skew 500 ppm
Sep 27 02:38:41 dist-test-slave-zv8z krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476721, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:38:41.501118 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:38:41.501991 1024 webserver.cc:460] Webserver started at http://127.0.98.1:42969/ using document root <none> and password file <none>
I20260927 02:38:41.502573 1024 fs_manager.cc:359] Metadata directory not provided
I20260927 02:38:41.502638 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:38:41.502851 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:38:41.504798 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/ts-0/data/instance:
uuid: "264c19780ce6489f8cfb14f6381cebdc"
format_stamp: "Formatted at 2026-09-27 02:38:41 on dist-test-slave-zv8z"
server_key: "1df88ec75757a2258705055800273d5c"
server_key_iv: "b69424875d30bbf9383bd5a8ff392fff"
server_key_version: "encryptionkey@0"
I20260927 02:38:41.505339 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/ts-0/wal/instance:
uuid: "264c19780ce6489f8cfb14f6381cebdc"
format_stamp: "Formatted at 2026-09-27 02:38:41 on dist-test-slave-zv8z"
server_key: "1df88ec75757a2258705055800273d5c"
server_key_iv: "b69424875d30bbf9383bd5a8ff392fff"
server_key_version: "encryptionkey@0"
I20260927 02:38:41.508906 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260927 02:38:41.511330 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:41.512583 1024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:38:41.512693 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/ts-0/wal
uuid: "264c19780ce6489f8cfb14f6381cebdc"
format_stamp: "Formatted at 2026-09-27 02:38:41 on dist-test-slave-zv8z"
server_key: "1df88ec75757a2258705055800273d5c"
server_key_iv: "b69424875d30bbf9383bd5a8ff392fff"
server_key_version: "encryptionkey@0"
I20260927 02:38:41.512822 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:38:41.542610 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:38:41.545723 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:38:41.545912 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:38:41.546535 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:38:41.547345 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:38:41.547394 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:41.547446 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:38:41.547474 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:41.557015 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35869
I20260927 02:38:41.557190 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35869 every 8 connection(s)
I20260927 02:38:41.557873 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:38:41.562098 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 1024
I20260927 02:38:41.562201 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:38:41.562493 392 external_mini_cluster.cc:1428] Setting key 37d2a4ed7d7d880fad2f2f722a0d1776
Sep 27 02:38:41 dist-test-slave-zv8z krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476721, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:38:41.571640 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:41.559750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40779 (local address 127.0.98.62:39585)
0927 02:38:41.559884 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:41.559888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:41.560686 (+ 798us) server_negotiation.cc:407] Connection header received
0927 02:38:41.561555 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:41.561559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:41.561612 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:41.561683 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:41.563246 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:41.563773 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:41.565270 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:41.565574 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:41.565671 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:41.568104 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:41.568108 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:41.568111 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:41.568138 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:41.569650 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:41.570120 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:41.570124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:41.570126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:41.570215 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:41.570554 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:41.570557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:41.570559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:41.570790 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:41.570901 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:41.571345 (+ 444us) server_negotiation.cc:299] Negotiation successful
0927 02:38:41.571468 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 02:38:41.572403 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:41.560111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39585 (local address 127.0.98.1:40779)
0927 02:38:41.560539 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:41.560579 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:41.561363 (+ 784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:41.561874 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:41.561883 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:41.562406 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:38:41.563084 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:41.563111 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:41.563883 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:41.563886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:41.565147 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:41.565159 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:41.565717 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:41.565720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:41.565876 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:41.566513 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:38:41.566534 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:38:41.567946 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:38:41.569777 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:41.569781 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:41.569794 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:41.570029 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:41.570336 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:41.570339 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:41.570340 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:41.570466 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:41.570890 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:38:41.570895 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:38:41.571117 (+ 222us) client_negotiation.cc:769] Sending connection context
0927 02:38:41.571276 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 02:38:41.571533 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":116,"threads_started":1}
I20260927 02:38:41.573637 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39585
I20260927 02:38:41.573863 1153 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 02:38:41.574421 1153 heartbeater.cc:511] Master 127.0.98.62:39585 requested a full tablet report, sending...
I20260927 02:38:41.576011 962 ts_manager.cc:194] Registered new tserver with Master: 264c19780ce6489f8cfb14f6381cebdc (127.0.98.1:35869)
I20260927 02:38:41.577167 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40779
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:38:41.588491 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39585
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:38:41.694736 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:38:41.695032 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:38:41.695078 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:38:41.699069 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:38:41.699124 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:38:41.699180 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:38:41.704248 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39585
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:38:41.705163 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:38:41.706076 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:38:41.715232 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:38:41.715210 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:38:41.715210 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:38:41.715346 1158 server_base.cc:1034] running on GCE node
I20260927 02:38:41.715903 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:38:41.716530 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:38:41.717707 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790476721717681 us; error 35 us; skew 500 ppm
Sep 27 02:38:41 dist-test-slave-zv8z krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476721, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:38:41.720395 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:38:41.721240 1158 webserver.cc:460] Webserver started at http://127.0.98.2:40963/ using document root <none> and password file <none>
I20260927 02:38:41.721736 1158 fs_manager.cc:359] Metadata directory not provided
I20260927 02:38:41.721789 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:38:41.721952 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:38:41.723917 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/ts-1/data/instance:
uuid: "2252d42250444c49a4337956b6a7f4c0"
format_stamp: "Formatted at 2026-09-27 02:38:41 on dist-test-slave-zv8z"
server_key: "b1fed4811c2ab02c85de8617abfa38f8"
server_key_iv: "645276151c9f14ba4f1599d1c94c61db"
server_key_version: "encryptionkey@0"
I20260927 02:38:41.724409 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2252d42250444c49a4337956b6a7f4c0"
format_stamp: "Formatted at 2026-09-27 02:38:41 on dist-test-slave-zv8z"
server_key: "b1fed4811c2ab02c85de8617abfa38f8"
server_key_iv: "645276151c9f14ba4f1599d1c94c61db"
server_key_version: "encryptionkey@0"
I20260927 02:38:41.727954 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:38:41.730208 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:41.731428 1158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:38:41.731518 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/ts-1/wal
uuid: "2252d42250444c49a4337956b6a7f4c0"
format_stamp: "Formatted at 2026-09-27 02:38:41 on dist-test-slave-zv8z"
server_key: "b1fed4811c2ab02c85de8617abfa38f8"
server_key_iv: "645276151c9f14ba4f1599d1c94c61db"
server_key_version: "encryptionkey@0"
I20260927 02:38:41.731594 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:38:41.757864 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:38:41.760941 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:38:41.761088 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:38:41.761605 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:38:41.762372 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:38:41.762420 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:41.762458 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:38:41.762471 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:41.770289 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46807
I20260927 02:38:41.770349 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46807 every 8 connection(s)
I20260927 02:38:41.771126 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:38:41.774109 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 1158
I20260927 02:38:41.774233 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:38:41.774549 392 external_mini_cluster.cc:1428] Setting key 9bd4feab36009a06aff4ac3d81d012d2
Sep 27 02:38:41 dist-test-slave-zv8z krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476721, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:38:41.785419 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:41.772924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37477 (local address 127.0.98.62:39585)
0927 02:38:41.773038 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:41.773042 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:41.773776 (+ 734us) server_negotiation.cc:407] Connection header received
0927 02:38:41.775410 (+ 1634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:41.775413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:41.775463 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:41.775538 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:41.776958 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:41.777473 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:41.779027 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:41.779242 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:41.779321 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:41.782036 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:41.782039 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:41.782041 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:41.782083 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:41.783510 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:41.784070 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:41.784073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:41.784074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:41.784146 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:41.784416 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:41.784418 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:41.784419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:41.784584 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:41.784653 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:41.785147 (+ 494us) server_negotiation.cc:299] Negotiation successful
0927 02:38:41.785245 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 02:38:41.786320 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:41.773224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39585 (local address 127.0.98.2:37477)
0927 02:38:41.773641 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:41.773680 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:41.775212 (+ 1532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:41.775749 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:41.775758 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:41.776181 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:38:41.776817 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:41.776834 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:41.777593 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:41.777597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:41.778922 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:41.778932 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:41.779380 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:41.779384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:41.779571 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:41.780129 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:38:41.780148 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:38:41.781854 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:38:41.783626 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:41.783633 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:41.783649 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:41.783980 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:41.784215 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:41.784217 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:41.784218 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:41.784338 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:41.784652 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:38:41.784658 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:38:41.784939 (+ 281us) client_negotiation.cc:769] Sending connection context
0927 02:38:41.785094 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 02:38:41.785362 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":120,"threads_started":1}
I20260927 02:38:41.787359 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39585
I20260927 02:38:41.787580 1287 heartbeater.cc:464] Registering TS with master...
I20260927 02:38:41.788054 1287 heartbeater.cc:511] Master 127.0.98.62:39585 requested a full tablet report, sending...
I20260927 02:38:41.789057 962 ts_manager.cc:194] Registered new tserver with Master: 2252d42250444c49a4337956b6a7f4c0 (127.0.98.2:46807)
I20260927 02:38:41.789753 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37477
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:38:41.804971 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39585
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:38:41.911307 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:38:41.911662 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:38:41.911737 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:38:41.915808 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:38:41.915869 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:38:41.915937 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:38:41.921339 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39585
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:38:41.922353 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:38:41.923301 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:38:41.932379 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:38:41.932386 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:38:41.932400 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:38:41.932767 1292 server_base.cc:1034] running on GCE node
I20260927 02:38:41.933173 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:38:41.933764 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:38:41.934917 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790476721934900 us; error 30 us; skew 500 ppm
Sep 27 02:38:41 dist-test-slave-zv8z krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476721, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:38:41.937493 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:38:41.938396 1292 webserver.cc:460] Webserver started at http://127.0.98.3:36355/ using document root <none> and password file <none>
I20260927 02:38:41.938900 1292 fs_manager.cc:359] Metadata directory not provided
I20260927 02:38:41.938956 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:38:41.939131 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:38:41.941074 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/ts-2/data/instance:
uuid: "5e58acf79d8f4d05af84c9267ddbc30a"
format_stamp: "Formatted at 2026-09-27 02:38:41 on dist-test-slave-zv8z"
server_key: "4cbb52bbdd740b14b957539ab8bf08a1"
server_key_iv: "7b95e57fe5eacd604d9a24e9f956e1ba"
server_key_version: "encryptionkey@0"
I20260927 02:38:41.941578 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5e58acf79d8f4d05af84c9267ddbc30a"
format_stamp: "Formatted at 2026-09-27 02:38:41 on dist-test-slave-zv8z"
server_key: "4cbb52bbdd740b14b957539ab8bf08a1"
server_key_iv: "7b95e57fe5eacd604d9a24e9f956e1ba"
server_key_version: "encryptionkey@0"
I20260927 02:38:41.945102 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 02:38:41.947419 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:41.948501 1292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:38:41.948599 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/ts-2/wal
uuid: "5e58acf79d8f4d05af84c9267ddbc30a"
format_stamp: "Formatted at 2026-09-27 02:38:41 on dist-test-slave-zv8z"
server_key: "4cbb52bbdd740b14b957539ab8bf08a1"
server_key_iv: "7b95e57fe5eacd604d9a24e9f956e1ba"
server_key_version: "encryptionkey@0"
I20260927 02:38:41.948683 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:38:41.959694 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:38:41.962599 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:38:41.962755 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:38:41.963269 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:38:41.964064 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:38:41.964111 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:41.964161 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:38:41.964185 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:41.972170 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37699
I20260927 02:38:41.972229 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37699 every 8 connection(s)
I20260927 02:38:41.972997 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:38:41.980804 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 1292
I20260927 02:38:41.980921 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790476718023131-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:38:41.981176 392 external_mini_cluster.cc:1428] Setting key 66917891f75e213e937d79b09295228b
Sep 27 02:38:41 dist-test-slave-zv8z krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476721, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:38:41.985966 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:41.974649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40613 (local address 127.0.98.62:39585)
0927 02:38:41.974774 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:41.974777 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:41.975641 (+ 864us) server_negotiation.cc:407] Connection header received
0927 02:38:41.977056 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:41.977059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:41.977111 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:41.977184 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:41.978305 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:41.978831 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:41.979909 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:41.980120 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:41.980206 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:41.982866 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:41.982869 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:41.982871 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:41.982896 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:41.984154 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:41.984611 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:41.984615 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:41.984616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:41.984687 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:41.984992 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:41.984995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:41.984997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:41.985164 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:41.985244 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:41.985699 (+ 455us) server_negotiation.cc:299] Negotiation successful
0927 02:38:41.985814 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 02:38:41.986788 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:41.975040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39585 (local address 127.0.98.3:40613)
0927 02:38:41.975507 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:41.975549 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:41.976859 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:41.977359 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:41.977366 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:41.977695 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:38:41.978143 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:41.978155 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:41.978946 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:41.978948 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:41.979801 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:41.979809 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:41.980256 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:41.980261 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:41.980416 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:41.981042 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:38:41.981058 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:38:41.982710 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:38:41.984260 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:41.984266 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:41.984280 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:41.984520 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:41.984766 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:41.984770 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:41.984772 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:41.984909 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:41.985252 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:38:41.985258 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:38:41.985487 (+ 229us) client_negotiation.cc:769] Sending connection context
0927 02:38:41.985642 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 02:38:41.985951 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":115,"threads_started":1}
I20260927 02:38:41.987797 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39585
I20260927 02:38:41.987996 1421 heartbeater.cc:464] Registering TS with master...
I20260927 02:38:41.988415 1421 heartbeater.cc:511] Master 127.0.98.62:39585 requested a full tablet report, sending...
I20260927 02:38:41.989351 962 ts_manager.cc:194] Registered new tserver with Master: 5e58acf79d8f4d05af84c9267ddbc30a (127.0.98.3:37699)
I20260927 02:38:41.989868 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40613
I20260927 02:38:41.994711 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:38:42.004760 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:41.997828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51876 (local address 127.0.98.62:39585)
0927 02:38:41.997998 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:41.998003 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:41.998493 (+ 490us) server_negotiation.cc:407] Connection header received
0927 02:38:41.998603 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:41.998607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:41.998648 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:41.998712 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:41.999348 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:41.999854 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:42.000641 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:42.000868 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:42.000950 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:42.002130 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:42.002134 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:42.002137 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:42.002163 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:42.003298 (+ 1135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:42.003661 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:42.003665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:42.003667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:42.003743 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:42.003972 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:42.003976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:42.003977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:42.004178 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:42.004291 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:42.004509 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 02:38:42.004611 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 02:38:42.009083 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51876:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:38:42.012012 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:38:42.025949 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:42.020857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47832 (local address 127.0.98.3:37699)
0927 02:38:42.021539 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:42.021550 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:42.021569 (+ 19us) server_negotiation.cc:407] Connection header received
0927 02:38:42.021635 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:42.021640 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:42.021806 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:42.021944 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:38:42.022286 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:42.023396 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:42.024813 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:42.025348 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:42.025440 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:42.025581 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:42.025712 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 02:38:42.025798 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":127,"threads_started":1}
I20260927 02:38:42.025941 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:42.020239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37699 (local address 127.0.0.1:47832)
0927 02:38:42.021176 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:42.021405 (+ 229us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:42.021477 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:42.022057 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:42.022059 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:42.022074 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:38:42.022148 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:42.022152 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:42.023521 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:42.023524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:42.024717 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:42.024724 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:42.025451 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:42.025454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:42.025556 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:42.025589 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 02:38:42.025687 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 02:38:42.025798 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":840,"mutex_wait_us":37,"thread_start_us":103,"threads_started":1}
I20260927 02:38:42.026288 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:42.020103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46807 (local address 127.0.0.1:55692)
0927 02:38:42.020550 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:42.020595 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:42.020712 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:42.021911 (+ 1199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:42.021917 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:42.021939 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:38:42.022074 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:42.022082 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:42.023172 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:42.023175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:42.024471 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:42.024482 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:42.025667 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:42.025671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:42.025830 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:42.025852 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 02:38:42.025920 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 02:38:42.025986 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":4,"thread_start_us":71,"threads_started":1}
I20260927 02:38:42.026758 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:42.020794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35869 (local address 127.0.0.1:47298)
0927 02:38:42.021203 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:42.021214 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:42.021294 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:42.022784 (+ 1490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:42.022787 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:42.022805 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:38:42.022892 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:42.022898 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:42.024126 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:42.024129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:42.025369 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:42.025379 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:42.026426 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:42.026430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:42.026512 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:42.026533 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 02:38:42.026576 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 02:38:42.026638 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":31,"thread_start_us":88,"threads_started":1}
I20260927 02:38:42.026921 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:42.020238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55692 (local address 127.0.98.2:46807)
0927 02:38:42.021437 (+ 1199us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:42.021449 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:42.021471 (+ 22us) server_negotiation.cc:407] Connection header received
0927 02:38:42.021549 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:42.021553 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:42.021738 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:42.021906 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:38:42.022182 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:42.023074 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:42.024575 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:42.025399 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:42.025484 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:42.025613 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:42.026591 (+ 978us) server_negotiation.cc:299] Negotiation successful
0927 02:38:42.026678 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1102,"thread_start_us":131,"threads_started":1}
I20260927 02:38:42.027544 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:42.021038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47298 (local address 127.0.98.1:35869)
0927 02:38:42.022393 (+ 1355us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:42.022406 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:42.022425 (+ 19us) server_negotiation.cc:407] Connection header received
0927 02:38:42.022489 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:42.022493 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:42.022652 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:42.022804 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:38:42.022982 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:42.023992 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:42.025508 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:42.026298 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:42.026919 (+ 621us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:42.027285 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:42.027329 (+ 44us) server_negotiation.cc:299] Negotiation successful
0927 02:38:42.027406 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1264,"thread_start_us":91,"threads_started":1}
I20260927 02:38:42.028759 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 93210853fef342eeaab93af65517e4a1 (DEFAULT_TABLE table=test-table [id=f6aee3267f324ef190ee11894557d3e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:38:42.029172 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 93210853fef342eeaab93af65517e4a1 (DEFAULT_TABLE table=test-table [id=f6aee3267f324ef190ee11894557d3e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:38:42.029774 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93210853fef342eeaab93af65517e4a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:38:42.029991 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93210853fef342eeaab93af65517e4a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:38:42.032785 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 93210853fef342eeaab93af65517e4a1 (DEFAULT_TABLE table=test-table [id=f6aee3267f324ef190ee11894557d3e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:38:42.033751 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93210853fef342eeaab93af65517e4a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:38:42.036605 1439 tablet_bootstrap.cc:492] T 93210853fef342eeaab93af65517e4a1 P 2252d42250444c49a4337956b6a7f4c0: Bootstrap starting.
I20260927 02:38:42.038582 1438 tablet_bootstrap.cc:492] T 93210853fef342eeaab93af65517e4a1 P 5e58acf79d8f4d05af84c9267ddbc30a: Bootstrap starting.
I20260927 02:38:42.038727 1439 tablet_bootstrap.cc:654] T 93210853fef342eeaab93af65517e4a1 P 2252d42250444c49a4337956b6a7f4c0: Neither blocks nor log segments found. Creating new log.
I20260927 02:38:42.039813 1439 log.cc:824] T 93210853fef342eeaab93af65517e4a1 P 2252d42250444c49a4337956b6a7f4c0: Log is configured to *not* fsync() on all Append() calls
I20260927 02:38:42.040230 1440 tablet_bootstrap.cc:492] T 93210853fef342eeaab93af65517e4a1 P 264c19780ce6489f8cfb14f6381cebdc: Bootstrap starting.
I20260927 02:38:42.040673 1438 tablet_bootstrap.cc:654] T 93210853fef342eeaab93af65517e4a1 P 5e58acf79d8f4d05af84c9267ddbc30a: Neither blocks nor log segments found. Creating new log.
I20260927 02:38:42.041867 1438 log.cc:824] T 93210853fef342eeaab93af65517e4a1 P 5e58acf79d8f4d05af84c9267ddbc30a: Log is configured to *not* fsync() on all Append() calls
I20260927 02:38:42.042066 1440 tablet_bootstrap.cc:654] T 93210853fef342eeaab93af65517e4a1 P 264c19780ce6489f8cfb14f6381cebdc: Neither blocks nor log segments found. Creating new log.
I20260927 02:38:42.042413 1439 tablet_bootstrap.cc:492] T 93210853fef342eeaab93af65517e4a1 P 2252d42250444c49a4337956b6a7f4c0: No bootstrap required, opened a new log
I20260927 02:38:42.042558 1439 ts_tablet_manager.cc:1397] T 93210853fef342eeaab93af65517e4a1 P 2252d42250444c49a4337956b6a7f4c0: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 02:38:42.043314 1440 log.cc:824] T 93210853fef342eeaab93af65517e4a1 P 264c19780ce6489f8cfb14f6381cebdc: Log is configured to *not* fsync() on all Append() calls
I20260927 02:38:42.044386 1438 tablet_bootstrap.cc:492] T 93210853fef342eeaab93af65517e4a1 P 5e58acf79d8f4d05af84c9267ddbc30a: No bootstrap required, opened a new log
I20260927 02:38:42.044571 1438 ts_tablet_manager.cc:1397] T 93210853fef342eeaab93af65517e4a1 P 5e58acf79d8f4d05af84c9267ddbc30a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 02:38:42.044857 1440 tablet_bootstrap.cc:492] T 93210853fef342eeaab93af65517e4a1 P 264c19780ce6489f8cfb14f6381cebdc: No bootstrap required, opened a new log
I20260927 02:38:42.045014 1440 ts_tablet_manager.cc:1397] T 93210853fef342eeaab93af65517e4a1 P 264c19780ce6489f8cfb14f6381cebdc: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 02:38:42.046663 1439 raft_consensus.cc:348] T 93210853fef342eeaab93af65517e4a1 P 2252d42250444c49a4337956b6a7f4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2252d42250444c49a4337956b6a7f4c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46807 } } peers { permanent_uuid: "5e58acf79d8f4d05af84c9267ddbc30a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37699 } } peers { permanent_uuid: "264c19780ce6489f8cfb14f6381cebdc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35869 } }
I20260927 02:38:42.046940 1439 raft_consensus.cc:374] T 93210853fef342eeaab93af65517e4a1 P 2252d42250444c49a4337956b6a7f4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:38:42.046995 1439 raft_consensus.cc:729] T 93210853fef342eeaab93af65517e4a1 P 2252d42250444c49a4337956b6a7f4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2252d42250444c49a4337956b6a7f4c0, State: Initialized, Role: FOLLOWER
I20260927 02:38:42.047470 1439 consensus_queue.cc:260] T 93210853fef342eeaab93af65517e4a1 P 2252d42250444c49a4337956b6a7f4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2252d42250444c49a4337956b6a7f4c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46807 } } peers { permanent_uuid: "5e58acf79d8f4d05af84c9267ddbc30a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37699 } } peers { permanent_uuid: "264c19780ce6489f8cfb14f6381cebdc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35869 } }
I20260927 02:38:42.047740 1440 raft_consensus.cc:348] T 93210853fef342eeaab93af65517e4a1 P 264c19780ce6489f8cfb14f6381cebdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2252d42250444c49a4337956b6a7f4c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46807 } } peers { permanent_uuid: "5e58acf79d8f4d05af84c9267ddbc30a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37699 } } peers { permanent_uuid: "264c19780ce6489f8cfb14f6381cebdc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35869 } }
I20260927 02:38:42.047971 1440 raft_consensus.cc:374] T 93210853fef342eeaab93af65517e4a1 P 264c19780ce6489f8cfb14f6381cebdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:38:42.048019 1440 raft_consensus.cc:729] T 93210853fef342eeaab93af65517e4a1 P 264c19780ce6489f8cfb14f6381cebdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 264c19780ce6489f8cfb14f6381cebdc, State: Initialized, Role: FOLLOWER
I20260927 02:38:42.048183 1438 raft_consensus.cc:348] T 93210853fef342eeaab93af65517e4a1 P 5e58acf79d8f4d05af84c9267ddbc30a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2252d42250444c49a4337956b6a7f4c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46807 } } peers { permanent_uuid: "5e58acf79d8f4d05af84c9267ddbc30a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37699 } } peers { permanent_uuid: "264c19780ce6489f8cfb14f6381cebdc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35869 } }
I20260927 02:38:42.048405 1438 raft_consensus.cc:374] T 93210853fef342eeaab93af65517e4a1 P 5e58acf79d8f4d05af84c9267ddbc30a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:38:42.048439 1439 ts_tablet_manager.cc:1428] T 93210853fef342eeaab93af65517e4a1 P 2252d42250444c49a4337956b6a7f4c0: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 02:38:42.048458 1438 raft_consensus.cc:729] T 93210853fef342eeaab93af65517e4a1 P 5e58acf79d8f4d05af84c9267ddbc30a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e58acf79d8f4d05af84c9267ddbc30a, State: Initialized, Role: FOLLOWER
I20260927 02:38:42.048460 1440 consensus_queue.cc:260] T 93210853fef342eeaab93af65517e4a1 P 264c19780ce6489f8cfb14f6381cebdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2252d42250444c49a4337956b6a7f4c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46807 } } peers { permanent_uuid: "5e58acf79d8f4d05af84c9267ddbc30a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37699 } } peers { permanent_uuid: "264c19780ce6489f8cfb14f6381cebdc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35869 } }
I20260927 02:38:42.048774 1287 heartbeater.cc:503] Master 127.0.98.62:39585 was elected leader, sending a full tablet report...
I20260927 02:38:42.048894 1438 consensus_queue.cc:260] T 93210853fef342eeaab93af65517e4a1 P 5e58acf79d8f4d05af84c9267ddbc30a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2252d42250444c49a4337956b6a7f4c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46807 } } peers { permanent_uuid: "5e58acf79d8f4d05af84c9267ddbc30a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37699 } } peers { permanent_uuid: "264c19780ce6489f8cfb14f6381cebdc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35869 } }
I20260927 02:38:42.049163 1153 heartbeater.cc:503] Master 127.0.98.62:39585 was elected leader, sending a full tablet report...
I20260927 02:38:42.049355 1440 ts_tablet_manager.cc:1428] T 93210853fef342eeaab93af65517e4a1 P 264c19780ce6489f8cfb14f6381cebdc: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260927 02:38:42.049779 1438 ts_tablet_manager.cc:1428] T 93210853fef342eeaab93af65517e4a1 P 5e58acf79d8f4d05af84c9267ddbc30a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 02:38:42.050146 1421 heartbeater.cc:503] Master 127.0.98.62:39585 was elected leader, sending a full tablet report...
W20260927 02:38:42.058964 1154 tablet.cc:2369] T 93210853fef342eeaab93af65517e4a1 P 264c19780ce6489f8cfb14f6381cebdc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:38:42.136868 1445 raft_consensus.cc:482] T 93210853fef342eeaab93af65517e4a1 P 264c19780ce6489f8cfb14f6381cebdc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:38:42.137106 1445 raft_consensus.cc:504] T 93210853fef342eeaab93af65517e4a1 P 264c19780ce6489f8cfb14f6381cebdc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2252d42250444c49a4337956b6a7f4c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46807 } } peers { permanent_uuid: "5e58acf79d8f4d05af84c9267ddbc30a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37699 } } peers { permanent_uuid: "264c19780ce6489f8cfb14f6381cebdc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35869 } }
I20260927 02:38:42.138366 1445 leader_election.cc:288] T 93210853fef342eeaab93af65517e4a1 P 264c19780ce6489f8cfb14f6381cebdc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2252d42250444c49a4337956b6a7f4c0 (127.0.98.2:46807), 5e58acf79d8f4d05af84c9267ddbc30a (127.0.98.3:37699)
I20260927 02:38:42.142283 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:42.138736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42439 (local address 127.0.98.2:46807)
0927 02:38:42.138905 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:42.138910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:42.138929 (+ 19us) server_negotiation.cc:407] Connection header received
0927 02:38:42.139073 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:42.139076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:42.139122 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:42.139189 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:38:42.139485 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:42.140091 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:42.141307 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:42.141755 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:42.141832 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:42.141890 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:42.142077 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 02:38:42.142143 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 02:38:42.142292 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:42.138515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46807 (local address 127.0.98.1:42439)
0927 02:38:42.138798 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:42.138814 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:42.138921 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:42.139235 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:42.139238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:42.139255 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:38:42.139369 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:42.139375 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:42.140212 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:42.140216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:42.141191 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:42.141201 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:42.141868 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:42.141872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:42.141965 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:42.141991 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 02:38:42.142057 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 02:38:42.142129 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":108,"threads_started":1}
I20260927 02:38:42.142400 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:42.138799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48915 (local address 127.0.98.3:37699)
0927 02:38:42.138956 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:42.138960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:42.139317 (+ 357us) server_negotiation.cc:407] Connection header received
0927 02:38:42.139442 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:42.139446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:42.139501 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:42.139558 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:38:42.139892 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:42.140471 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:42.141470 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:42.141893 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:42.141985 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:42.142038 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:42.142218 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 02:38:42.142299 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 02:38:42.142767 1448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:42.138515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37699 (local address 127.0.98.1:48915)
0927 02:38:42.139244 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:42.139255 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:42.139338 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:42.139588 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:42.139590 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:42.139606 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:38:42.139721 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:42.139727 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:42.140574 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:42.140576 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:42.141361 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:42.141371 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:42.141998 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:42.142001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:42.142107 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:42.142130 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 02:38:42.142198 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 02:38:42.142284 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"mutex_wait_us":430,"thread_start_us":112,"threads_started":1}
I20260927 02:38:42.142817 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93210853fef342eeaab93af65517e4a1" candidate_uuid: "264c19780ce6489f8cfb14f6381cebdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2252d42250444c49a4337956b6a7f4c0" is_pre_election: true
I20260927 02:38:42.143148 1241 raft_consensus.cc:2454] T 93210853fef342eeaab93af65517e4a1 P 2252d42250444c49a4337956b6a7f4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 264c19780ce6489f8cfb14f6381cebdc in term 0.
I20260927 02:38:42.143330 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93210853fef342eeaab93af65517e4a1" candidate_uuid: "264c19780ce6489f8cfb14f6381cebdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e58acf79d8f4d05af84c9267ddbc30a" is_pre_election: true
I20260927 02:38:42.143589 1040 leader_election.cc:302] T 93210853fef342eeaab93af65517e4a1 P 264c19780ce6489f8cfb14f6381cebdc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2252d42250444c49a4337956b6a7f4c0, 264c19780ce6489f8cfb14f6381cebdc; no voters:
I20260927 02:38:42.143608 1375 raft_consensus.cc:2454] T 93210853fef342eeaab93af65517e4a1 P 5e58acf79d8f4d05af84c9267ddbc30a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 264c19780ce6489f8cfb14f6381cebdc in term 0.
I20260927 02:38:42.143821 1445 raft_consensus.cc:2781] T 93210853fef342eeaab93af65517e4a1 P 264c19780ce6489f8cfb14f6381cebdc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:38:42.143903 1445 raft_consensus.cc:482] T 93210853fef342eeaab93af65517e4a1 P 264c19780ce6489f8cfb14f6381cebdc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:38:42.143944 1445 raft_consensus.cc:3037] T 93210853fef342eeaab93af65517e4a1 P 264c19780ce6489f8cfb14f6381cebdc [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:38:42.144927 1445 raft_consensus.cc:504] T 93210853fef342eeaab93af65517e4a1 P 264c19780ce6489f8cfb14f6381cebdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2252d42250444c49a4337956b6a7f4c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46807 } } peers { permanent_uuid: "5e58acf79d8f4d05af84c9267ddbc30a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37699 } } peers { permanent_uuid: "264c19780ce6489f8cfb14f6381cebdc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35869 } }
I20260927 02:38:42.145251 1445 leader_election.cc:288] T 93210853fef342eeaab93af65517e4a1 P 264c19780ce6489f8cfb14f6381cebdc [CANDIDATE]: Term 1 election: Requested vote from peers 2252d42250444c49a4337956b6a7f4c0 (127.0.98.2:46807), 5e58acf79d8f4d05af84c9267ddbc30a (127.0.98.3:37699)
I20260927 02:38:42.145537 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93210853fef342eeaab93af65517e4a1" candidate_uuid: "264c19780ce6489f8cfb14f6381cebdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2252d42250444c49a4337956b6a7f4c0"
I20260927 02:38:42.145545 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93210853fef342eeaab93af65517e4a1" candidate_uuid: "264c19780ce6489f8cfb14f6381cebdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e58acf79d8f4d05af84c9267ddbc30a"
I20260927 02:38:42.145638 1241 raft_consensus.cc:3037] T 93210853fef342eeaab93af65517e4a1 P 2252d42250444c49a4337956b6a7f4c0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:38:42.145643 1375 raft_consensus.cc:3037] T 93210853fef342eeaab93af65517e4a1 P 5e58acf79d8f4d05af84c9267ddbc30a [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:38:42.146653 1241 raft_consensus.cc:2454] T 93210853fef342eeaab93af65517e4a1 P 2252d42250444c49a4337956b6a7f4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 264c19780ce6489f8cfb14f6381cebdc in term 1.
I20260927 02:38:42.146914 1040 leader_election.cc:302] T 93210853fef342eeaab93af65517e4a1 P 264c19780ce6489f8cfb14f6381cebdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2252d42250444c49a4337956b6a7f4c0, 264c19780ce6489f8cfb14f6381cebdc; no voters:
I20260927 02:38:42.147084 1445 raft_consensus.cc:2781] T 93210853fef342eeaab93af65517e4a1 P 264c19780ce6489f8cfb14f6381cebdc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:38:42.147161 1375 raft_consensus.cc:2454] T 93210853fef342eeaab93af65517e4a1 P 5e58acf79d8f4d05af84c9267ddbc30a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 264c19780ce6489f8cfb14f6381cebdc in term 1.
I20260927 02:38:42.147287 1445 raft_consensus.cc:686] T 93210853fef342eeaab93af65517e4a1 P 264c19780ce6489f8cfb14f6381cebdc [term 1 LEADER]: Becoming Leader. State: Replica: 264c19780ce6489f8cfb14f6381cebdc, State: Running, Role: LEADER
I20260927 02:38:42.147621 1445 consensus_queue.cc:237] T 93210853fef342eeaab93af65517e4a1 P 264c19780ce6489f8cfb14f6381cebdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2252d42250444c49a4337956b6a7f4c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46807 } } peers { permanent_uuid: "5e58acf79d8f4d05af84c9267ddbc30a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37699 } } peers { permanent_uuid: "264c19780ce6489f8cfb14f6381cebdc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35869 } }
I20260927 02:38:42.150540 962 catalog_manager.cc:5495] T 93210853fef342eeaab93af65517e4a1 P 264c19780ce6489f8cfb14f6381cebdc reported cstate change: term changed from 0 to 1, leader changed from <none> to 264c19780ce6489f8cfb14f6381cebdc (127.0.98.1). New cstate: current_term: 1 leader_uuid: "264c19780ce6489f8cfb14f6381cebdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2252d42250444c49a4337956b6a7f4c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e58acf79d8f4d05af84c9267ddbc30a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "264c19780ce6489f8cfb14f6381cebdc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35869 } health_report { overall_health: HEALTHY } } }
Sep 27 02:38:42 dist-test-slave-zv8z krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:38:42.178725 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:42.170026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47310 (local address 127.0.98.1:35869)
0927 02:38:42.170176 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:42.170179 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:42.170230 (+ 51us) server_negotiation.cc:407] Connection header received
0927 02:38:42.170370 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:42.170372 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:42.170412 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:42.170479 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:42.171144 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:42.171658 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:42.172436 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:42.172653 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:42.172739 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:42.175294 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:42.175305 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:42.175320 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:42.175345 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:42.177023 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:42.177435 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:42.177441 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:42.177442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:42.177514 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:42.177839 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:42.177843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:42.177844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:42.178095 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:42.178299 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:42.178496 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 02:38:42.178599 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 02:38:42 dist-test-slave-zv8z krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476722, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 02:38:42 dist-test-slave-zv8z krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476722, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:38:42.204368 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:42.196091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47320 (local address 127.0.98.1:35869)
0927 02:38:42.196251 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:42.196255 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:42.196347 (+ 92us) server_negotiation.cc:407] Connection header received
0927 02:38:42.196520 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:42.196524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:42.196568 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:42.196639 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:42.197323 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:42.197832 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:42.198686 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:42.198910 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:42.198998 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:42.201364 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:42.201367 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:42.201369 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:42.201414 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:42.202720 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:42.203200 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:42.203202 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:42.203204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:42.203288 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:42.203539 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:42.203541 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:42.203543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:42.203695 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:42.203815 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:42.204131 (+ 316us) server_negotiation.cc:299] Negotiation successful
0927 02:38:42.204245 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 02:38:42 dist-test-slave-zv8z krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260927 02:38:42.223874 1422 tablet.cc:2369] T 93210853fef342eeaab93af65517e4a1 P 5e58acf79d8f4d05af84c9267ddbc30a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 02:38:42 dist-test-slave-zv8z krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:38:42.228554 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:42.220077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47334 (local address 127.0.98.1:35869)
0927 02:38:42.220227 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:42.220231 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:42.220311 (+ 80us) server_negotiation.cc:407] Connection header received
0927 02:38:42.220451 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:42.220454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:42.220494 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:42.220552 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:42.221285 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:42.221845 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:42.222837 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:42.223056 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:42.223139 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:42.225694 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:42.225698 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:42.225699 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:42.225725 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:42.226987 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:42.227442 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:42.227445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:42.227446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:42.227539 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:42.227788 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:42.227791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:42.227793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:42.227958 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:42.228087 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:42.228290 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 02:38:42.228403 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260927 02:38:42.229838 1060 tablet_service.cc:2908] Rejecting scan request for tablet 93210853fef342eeaab93af65517e4a1: Uninitialized: clean time has not yet been initialized
I20260927 02:38:42.231922 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 1024
I20260927 02:38:42.237692 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 1158
I20260927 02:38:42.242141 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 1292
I20260927 02:38:42.246932 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 930
2026-09-27T02:38:42Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2186 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:38:42 dist-test-slave-zv8z krb5kdc[1463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:38:42 dist-test-slave-zv8z krb5kdc[1463](info): set up 2 sockets
Sep 27 02:38:42 dist-test-slave-zv8z krb5kdc[1463](info): commencing operation
krb5kdc: starting...
W20260927 02:38:43.197832 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/krb5kdc/test-user.keytab.
Sep 27 02:38:43 dist-test-slave-zv8z krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:38:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:38:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:38:43.311508 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40591
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:38:43.415493 1476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:38:43.415751 1476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:38:43.415789 1476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:38:43.419652 1476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:38:43.419696 1476 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 02:38:43.419711 1476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:38:43.419726 1476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:38:43.419741 1476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:38:43.419754 1476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:38:43.424841 1476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40591
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:38:43.425819 1476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:38:43.426805 1476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:38:43.435070 1482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:38:43.435087 1481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:38:43.435170 1476 server_base.cc:1034] running on GCE node
W20260927 02:38:43.435232 1484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:38:43.435840 1476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:38:43.436885 1476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:38:43.438041 1476 hybrid_clock.cc:648] HybridClock initialized: now 1790476723438024 us; error 28 us; skew 500 ppm
Sep 27 02:38:43 dist-test-slave-zv8z krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476723, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:38:43.444342 1476 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:38:43.445168 1476 webserver.cc:460] Webserver started at http://127.0.98.62:33283/ using document root <none> and password file <none>
I20260927 02:38:43.445668 1476 fs_manager.cc:359] Metadata directory not provided
I20260927 02:38:43.445724 1476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:38:43.445884 1476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:38:43.447839 1476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/master-0/data/instance:
uuid: "edff1a4c13434b2e9d0ddfc6409df266"
format_stamp: "Formatted at 2026-09-27 02:38:43 on dist-test-slave-zv8z"
server_key: "941a18914798a47acdd54aba59d24ca5"
server_key_iv: "00a3f3695572dbab186b0135f6649321"
server_key_version: "encryptionkey@0"
I20260927 02:38:43.448318 1476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/master-0/wal/instance:
uuid: "edff1a4c13434b2e9d0ddfc6409df266"
format_stamp: "Formatted at 2026-09-27 02:38:43 on dist-test-slave-zv8z"
server_key: "941a18914798a47acdd54aba59d24ca5"
server_key_iv: "00a3f3695572dbab186b0135f6649321"
server_key_version: "encryptionkey@0"
I20260927 02:38:43.451704 1476 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260927 02:38:43.453867 1491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:43.455020 1476 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:38:43.455128 1476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "edff1a4c13434b2e9d0ddfc6409df266"
format_stamp: "Formatted at 2026-09-27 02:38:43 on dist-test-slave-zv8z"
server_key: "941a18914798a47acdd54aba59d24ca5"
server_key_iv: "00a3f3695572dbab186b0135f6649321"
server_key_version: "encryptionkey@0"
I20260927 02:38:43.455217 1476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:38:43.474917 1476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:38:43.477591 1476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:38:43.477747 1476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:38:43.484800 1476 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40591
I20260927 02:38:43.484819 1553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40591 every 8 connection(s)
I20260927 02:38:43.485709 1476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:38:43.486752 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 1476
I20260927 02:38:43.486837 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/master-0/wal/instance
I20260927 02:38:43.487097 392 external_mini_cluster.cc:1428] Setting key be3032bb6db28e50e7ff609073f8668f
I20260927 02:38:43.488528 1554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:38:43.495009 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edff1a4c13434b2e9d0ddfc6409df266: Bootstrap starting.
Sep 27 02:38:43 dist-test-slave-zv8z krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:38:43.497375 1554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P edff1a4c13434b2e9d0ddfc6409df266: Neither blocks nor log segments found. Creating new log.
I20260927 02:38:43.498654 1554 log.cc:824] T 00000000000000000000000000000000 P edff1a4c13434b2e9d0ddfc6409df266: Log is configured to *not* fsync() on all Append() calls
I20260927 02:38:43.500435 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edff1a4c13434b2e9d0ddfc6409df266: No bootstrap required, opened a new log
I20260927 02:38:43.501087 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:43.488408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33744 (local address 127.0.98.62:40591)
0927 02:38:43.488801 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:43.488818 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:43.488859 (+ 41us) server_negotiation.cc:407] Connection header received
0927 02:38:43.489678 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:43.489700 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:43.490095 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:43.490460 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:43.491049 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:43.492509 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:43.493344 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:43.494055 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:43.494208 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:43.496494 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:43.496500 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:43.496515 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:43.496546 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:43.498431 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:43.498855 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:43.498860 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:43.498865 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:43.498974 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:43.499271 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:43.499274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:43.499276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:43.499646 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:43.499812 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:43.500101 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 02:38:43.500354 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":111,"threads_started":1}
I20260927 02:38:43.503697 1554 raft_consensus.cc:348] T 00000000000000000000000000000000 P edff1a4c13434b2e9d0ddfc6409df266 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edff1a4c13434b2e9d0ddfc6409df266" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40591 } }
I20260927 02:38:43.503904 1554 raft_consensus.cc:374] T 00000000000000000000000000000000 P edff1a4c13434b2e9d0ddfc6409df266 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:38:43.503968 1554 raft_consensus.cc:729] T 00000000000000000000000000000000 P edff1a4c13434b2e9d0ddfc6409df266 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edff1a4c13434b2e9d0ddfc6409df266, State: Initialized, Role: FOLLOWER
I20260927 02:38:43.504345 1554 consensus_queue.cc:260] T 00000000000000000000000000000000 P edff1a4c13434b2e9d0ddfc6409df266 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edff1a4c13434b2e9d0ddfc6409df266" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40591 } }
I20260927 02:38:43.504464 1554 raft_consensus.cc:388] T 00000000000000000000000000000000 P edff1a4c13434b2e9d0ddfc6409df266 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:38:43.504524 1554 raft_consensus.cc:482] T 00000000000000000000000000000000 P edff1a4c13434b2e9d0ddfc6409df266 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:38:43.504616 1554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edff1a4c13434b2e9d0ddfc6409df266 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:38:43.505456 1554 raft_consensus.cc:504] T 00000000000000000000000000000000 P edff1a4c13434b2e9d0ddfc6409df266 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edff1a4c13434b2e9d0ddfc6409df266" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40591 } }
I20260927 02:38:43.505762 1554 leader_election.cc:302] T 00000000000000000000000000000000 P edff1a4c13434b2e9d0ddfc6409df266 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: edff1a4c13434b2e9d0ddfc6409df266; no voters:
I20260927 02:38:43.506024 1554 leader_election.cc:288] T 00000000000000000000000000000000 P edff1a4c13434b2e9d0ddfc6409df266 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:38:43.506147 1559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P edff1a4c13434b2e9d0ddfc6409df266 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:38:43.506381 1559 raft_consensus.cc:686] T 00000000000000000000000000000000 P edff1a4c13434b2e9d0ddfc6409df266 [term 1 LEADER]: Becoming Leader. State: Replica: edff1a4c13434b2e9d0ddfc6409df266, State: Running, Role: LEADER
I20260927 02:38:43.506704 1559 consensus_queue.cc:237] T 00000000000000000000000000000000 P edff1a4c13434b2e9d0ddfc6409df266 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edff1a4c13434b2e9d0ddfc6409df266" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40591 } }
I20260927 02:38:43.507266 1554 sys_catalog.cc:564] T 00000000000000000000000000000000 P edff1a4c13434b2e9d0ddfc6409df266 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:38:43.508390 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P edff1a4c13434b2e9d0ddfc6409df266 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "edff1a4c13434b2e9d0ddfc6409df266" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edff1a4c13434b2e9d0ddfc6409df266" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40591 } } }
I20260927 02:38:43.508419 1561 sys_catalog.cc:455] T 00000000000000000000000000000000 P edff1a4c13434b2e9d0ddfc6409df266 [sys.catalog]: SysCatalogTable state changed. Reason: New leader edff1a4c13434b2e9d0ddfc6409df266. Latest consensus state: current_term: 1 leader_uuid: "edff1a4c13434b2e9d0ddfc6409df266" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edff1a4c13434b2e9d0ddfc6409df266" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40591 } } }
I20260927 02:38:43.508605 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P edff1a4c13434b2e9d0ddfc6409df266 [sys.catalog]: This master's current role is: LEADER
I20260927 02:38:43.508625 1561 sys_catalog.cc:458] T 00000000000000000000000000000000 P edff1a4c13434b2e9d0ddfc6409df266 [sys.catalog]: This master's current role is: LEADER
I20260927 02:38:43.508977 1570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:38:43.511616 1570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:38:43.517571 1570 catalog_manager.cc:1347] Generated new cluster ID: 51b2201376d94b9ea34506bfe730c67b
I20260927 02:38:43.517644 1570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:38:43.527637 1570 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:38:43.528828 1570 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:38:43.537837 1570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P edff1a4c13434b2e9d0ddfc6409df266: Generated new TSK 0
I20260927 02:38:43.538667 1570 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:38:43.567631 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40591
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:38:43.677063 1580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:38:43.677302 1580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:38:43.677361 1580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:38:43.681345 1580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:38:43.681397 1580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:38:43.681452 1580 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:38:43.681504 1580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:38:43.686542 1580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:38:43.687454 1580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:38:43.688370 1580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:38:43.697302 1586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:38:43.697402 1585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:38:43.697367 1588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:38:43.697487 1580 server_base.cc:1034] running on GCE node
I20260927 02:38:43.697969 1580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:38:43.698621 1580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:38:43.699774 1580 hybrid_clock.cc:648] HybridClock initialized: now 1790476723699747 us; error 39 us; skew 500 ppm
Sep 27 02:38:43 dist-test-slave-zv8z krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476723, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:38:43.702464 1580 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:38:43.703313 1580 webserver.cc:460] Webserver started at http://127.0.98.1:38951/ using document root <none> and password file <none>
I20260927 02:38:43.703819 1580 fs_manager.cc:359] Metadata directory not provided
I20260927 02:38:43.703876 1580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:38:43.704062 1580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:38:43.705962 1580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-0/data/instance:
uuid: "90f8e9f42b7c4601944a51a87c1cf4cd"
format_stamp: "Formatted at 2026-09-27 02:38:43 on dist-test-slave-zv8z"
server_key: "2b00c23d511360cc83ecb914102d68ad"
server_key_iv: "76c5fa2c3ffc593ad762cfb08f3e9f14"
server_key_version: "encryptionkey@0"
I20260927 02:38:43.706492 1580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-0/wal/instance:
uuid: "90f8e9f42b7c4601944a51a87c1cf4cd"
format_stamp: "Formatted at 2026-09-27 02:38:43 on dist-test-slave-zv8z"
server_key: "2b00c23d511360cc83ecb914102d68ad"
server_key_iv: "76c5fa2c3ffc593ad762cfb08f3e9f14"
server_key_version: "encryptionkey@0"
I20260927 02:38:43.709942 1580 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260927 02:38:43.712198 1595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:43.713322 1580 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:38:43.713411 1580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-0/wal
uuid: "90f8e9f42b7c4601944a51a87c1cf4cd"
format_stamp: "Formatted at 2026-09-27 02:38:43 on dist-test-slave-zv8z"
server_key: "2b00c23d511360cc83ecb914102d68ad"
server_key_iv: "76c5fa2c3ffc593ad762cfb08f3e9f14"
server_key_version: "encryptionkey@0"
I20260927 02:38:43.713505 1580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:38:43.733310 1580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:38:43.736177 1580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:38:43.736334 1580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:38:43.737672 1580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:38:43.737720 1580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:43.737757 1580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:38:43.737782 1580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 02:38:43 dist-test-slave-zv8z krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476723, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:38:43.747268 1580 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40523
I20260927 02:38:43.747334 1710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40523 every 8 connection(s)
I20260927 02:38:43.748266 1580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:38:43.750356 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:43.738362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45415 (local address 127.0.98.62:40591)
0927 02:38:43.738511 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:43.738517 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:43.739277 (+ 760us) server_negotiation.cc:407] Connection header received
0927 02:38:43.740326 (+ 1049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:43.740329 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:43.740383 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:43.740472 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:43.741962 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:43.742718 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:43.743917 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:43.744160 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:43.744246 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:43.746756 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:43.746760 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:43.746763 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:43.746791 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:43.748404 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:43.748912 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:43.748916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:43.748918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:43.749010 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:43.749313 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:43.749316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:43.749318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:43.749498 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:43.749581 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:43.750053 (+ 472us) server_negotiation.cc:299] Negotiation successful
0927 02:38:43.750174 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 02:38:43.751106 1604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:43.738717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40591 (local address 127.0.98.1:45415)
0927 02:38:43.739128 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:43.739174 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:43.740043 (+ 869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:43.740707 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:43.740716 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:43.741140 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:38:43.741795 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:43.741816 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:43.742840 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:43.742843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:43.743809 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:43.743818 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:43.744301 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:43.744305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:43.744445 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:43.744971 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:38:43.744987 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:38:43.746559 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:38:43.748504 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:43.748508 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:43.748518 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:43.748797 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:43.749086 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:43.749089 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:43.749091 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:43.749226 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:43.749593 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:38:43.749599 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:38:43.749826 (+ 227us) client_negotiation.cc:769] Sending connection context
0927 02:38:43.749992 (+ 166us) client_negotiation.cc:240] Negotiation successful
0927 02:38:43.750313 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":102,"threads_started":1}
I20260927 02:38:43.752161 1711 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40591
I20260927 02:38:43.752291 1711 heartbeater.cc:464] Registering TS with master...
I20260927 02:38:43.752719 1711 heartbeater.cc:511] Master 127.0.98.62:40591 requested a full tablet report, sending...
I20260927 02:38:43.753554 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 1580
I20260927 02:38:43.753654 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:38:43.753880 392 external_mini_cluster.cc:1428] Setting key 012ae8177b394ae6a9c6933e3a074287
I20260927 02:38:43.754228 1507 ts_manager.cc:194] Registered new tserver with Master: 90f8e9f42b7c4601944a51a87c1cf4cd (127.0.98.1:40523)
I20260927 02:38:43.755594 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45415
I20260927 02:38:43.761205 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:43.754121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40591 (local address 127.0.98.1:53833)
0927 02:38:43.754427 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:43.754440 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:43.754554 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:43.754762 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:43.754764 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:43.754935 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:38:43.755049 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:43.755056 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:43.755990 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:43.755994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:43.756780 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:43.756788 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:43.757206 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:43.757209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:43.757294 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:43.757737 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:38:43.757750 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:38:43.758133 (+ 383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:38:43.759920 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:43.759923 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:43.759925 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:43.760110 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:43.760386 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:43.760388 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:43.760390 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:43.760472 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:43.760833 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:38:43.760835 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:38:43.760900 (+ 65us) client_negotiation.cc:769] Sending connection context
0927 02:38:43.760970 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 02:38:43.761071 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":83,"threads_started":1}
I20260927 02:38:43.761296 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:43.754197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53833 (local address 127.0.98.62:40591)
0927 02:38:43.754313 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:43.754317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:43.754497 (+ 180us) server_negotiation.cc:407] Connection header received
0927 02:38:43.754632 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:43.754635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:43.754676 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:43.754741 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:43.755154 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:43.755858 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:43.756881 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:43.757109 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:43.757197 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:43.758234 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:43.758238 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:43.758240 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:43.758284 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:43.759824 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:43.760196 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:43.760200 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:43.760201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:43.760282 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:43.760553 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:43.760556 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:43.760558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:43.760757 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:43.760850 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:43.761037 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 02:38:43.761158 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:38:43.777376 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40591
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:38:43.881754 1720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:38:43.882012 1720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:38:43.882053 1720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:38:43.885972 1720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:38:43.886034 1720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:38:43.886068 1720 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:38:43.886153 1720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:38:43.891189 1720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:38:43.892185 1720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:38:43.893122 1720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:38:43.901872 1725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:38:43.902004 1720 server_base.cc:1034] running on GCE node
W20260927 02:38:43.901921 1726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:38:43.902175 1728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:38:43.902537 1720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:38:43.903084 1720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:38:43.904241 1720 hybrid_clock.cc:648] HybridClock initialized: now 1790476723904212 us; error 38 us; skew 500 ppm
Sep 27 02:38:43 dist-test-slave-zv8z krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476723, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:38:43.906745 1720 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:38:43.907598 1720 webserver.cc:460] Webserver started at http://127.0.98.2:42659/ using document root <none> and password file <none>
I20260927 02:38:43.908109 1720 fs_manager.cc:359] Metadata directory not provided
I20260927 02:38:43.908192 1720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:38:43.908422 1720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:38:43.910351 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-1/data/instance:
uuid: "428bd355e94941368a1b1f8c964d5bac"
format_stamp: "Formatted at 2026-09-27 02:38:43 on dist-test-slave-zv8z"
server_key: "749a4d6cf409f683df04b6a22f57ed4d"
server_key_iv: "cb8bdefa081ffeaf16cb5cd7da111b00"
server_key_version: "encryptionkey@0"
I20260927 02:38:43.910859 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-1/wal/instance:
uuid: "428bd355e94941368a1b1f8c964d5bac"
format_stamp: "Formatted at 2026-09-27 02:38:43 on dist-test-slave-zv8z"
server_key: "749a4d6cf409f683df04b6a22f57ed4d"
server_key_iv: "cb8bdefa081ffeaf16cb5cd7da111b00"
server_key_version: "encryptionkey@0"
I20260927 02:38:43.914304 1720 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 02:38:43.916481 1735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:43.917636 1720 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 02:38:43.917737 1720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-1/wal
uuid: "428bd355e94941368a1b1f8c964d5bac"
format_stamp: "Formatted at 2026-09-27 02:38:43 on dist-test-slave-zv8z"
server_key: "749a4d6cf409f683df04b6a22f57ed4d"
server_key_iv: "cb8bdefa081ffeaf16cb5cd7da111b00"
server_key_version: "encryptionkey@0"
I20260927 02:38:43.917829 1720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:38:43.945231 1720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:38:43.948093 1720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:38:43.948259 1720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:38:43.949584 1720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:38:43.949630 1720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:43.949666 1720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:38:43.949692 1720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 02:38:43 dist-test-slave-zv8z krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476723, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:38:43.959261 1720 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39007
I20260927 02:38:43.959303 1850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39007 every 8 connection(s)
I20260927 02:38:43.960269 1720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:38:43.962540 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:43.950461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35295 (local address 127.0.98.62:40591)
0927 02:38:43.950589 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:43.950594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:43.951504 (+ 910us) server_negotiation.cc:407] Connection header received
0927 02:38:43.952903 (+ 1399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:43.952907 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:43.952964 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:43.953054 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:43.954616 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:43.955246 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:43.956396 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:43.956638 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:43.956763 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:43.959220 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:43.959224 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:43.959227 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:43.959253 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:43.960665 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:43.961194 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:43.961198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:43.961200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:43.961270 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:43.961563 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:43.961566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:43.961568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:43.961735 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:43.961814 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:43.962281 (+ 467us) server_negotiation.cc:299] Negotiation successful
0927 02:38:43.962391 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 02:38:43.962886 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 1720
I20260927 02:38:43.962997 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:38:43.963271 392 external_mini_cluster.cc:1428] Setting key 5eb06746de23dca9f52e9c88057dc767
I20260927 02:38:43.963382 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:43.950866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40591 (local address 127.0.98.2:35295)
0927 02:38:43.951334 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:43.951394 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:43.952674 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:43.953233 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:43.953242 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:43.953767 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:38:43.954443 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:43.954463 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:43.955353 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:43.955357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:43.956285 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:43.956295 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:43.956790 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:43.956794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:43.956960 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:43.957448 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:38:43.957468 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:38:43.959030 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:38:43.960780 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:43.960785 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:43.960798 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:43.961084 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:43.961345 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:43.961348 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:43.961350 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:43.961473 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:43.961825 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:38:43.961830 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:38:43.962040 (+ 210us) client_negotiation.cc:769] Sending connection context
0927 02:38:43.962181 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 02:38:43.962483 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":114,"threads_started":1}
I20260927 02:38:43.964550 1851 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40591
I20260927 02:38:43.964707 1851 heartbeater.cc:464] Registering TS with master...
I20260927 02:38:43.965200 1851 heartbeater.cc:511] Master 127.0.98.62:40591 requested a full tablet report, sending...
I20260927 02:38:43.966158 1508 ts_manager.cc:194] Registered new tserver with Master: 428bd355e94941368a1b1f8c964d5bac (127.0.98.2:39007)
I20260927 02:38:43.966831 1508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35295
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:38:43.973265 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:43.966593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49247 (local address 127.0.98.62:40591)
0927 02:38:43.966687 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:43.966691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:43.966886 (+ 195us) server_negotiation.cc:407] Connection header received
0927 02:38:43.967023 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:43.967026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:43.967065 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:43.967135 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:43.967576 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:43.968268 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:43.969104 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:43.969429 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:43.969532 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:43.970503 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:43.970507 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:43.970510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:43.970535 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:43.971907 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:43.972264 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:43.972267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:43.972269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:43.972348 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:43.972545 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:43.972548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:43.972549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:43.972807 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:43.972903 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:43.973079 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 02:38:43.973163 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 02:38:43.973272 1858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:43.966519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40591 (local address 127.0.98.2:49247)
0927 02:38:43.966807 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:43.966820 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:43.966950 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:43.967164 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:43.967167 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:43.967353 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:38:43.967465 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:43.967472 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:43.968392 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:43.968395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:43.968999 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:43.969008 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:43.969523 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:43.969526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:43.969588 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:43.970016 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:38:43.970026 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:38:43.970382 (+ 356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:38:43.972001 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:43.972003 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:43.972005 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:43.972182 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:43.972419 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:43.972421 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:43.972423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:43.972480 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:43.972891 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:38:43.972893 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:38:43.972960 (+ 67us) client_negotiation.cc:769] Sending connection context
0927 02:38:43.973032 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 02:38:43.973124 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":88,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:38:43.987521 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40591
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:38:44.092020 1860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:38:44.092262 1860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:38:44.092303 1860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:38:44.096212 1860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:38:44.096267 1860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:38:44.096303 1860 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:38:44.096340 1860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:38:44.101440 1860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:38:44.102448 1860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:38:44.103394 1860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:38:44.112571 1866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:38:44.112735 1860 server_base.cc:1034] running on GCE node
W20260927 02:38:44.112592 1865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:38:44.112579 1868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:38:44.113235 1860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:38:44.113839 1860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:38:44.115012 1860 hybrid_clock.cc:648] HybridClock initialized: now 1790476724114994 us; error 35 us; skew 500 ppm
Sep 27 02:38:44 dist-test-slave-zv8z krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476724, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:38:44.117624 1860 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:38:44.118495 1860 webserver.cc:460] Webserver started at http://127.0.98.3:42105/ using document root <none> and password file <none>
I20260927 02:38:44.119007 1860 fs_manager.cc:359] Metadata directory not provided
I20260927 02:38:44.119061 1860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:38:44.119222 1860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:38:44.121152 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-2/data/instance:
uuid: "8da60306942c4204a3c8bc44eb76b7fb"
format_stamp: "Formatted at 2026-09-27 02:38:44 on dist-test-slave-zv8z"
server_key: "699312284469945900ed1e8cccf011d7"
server_key_iv: "28f49e80f19fef39f8b570d32da71572"
server_key_version: "encryptionkey@0"
I20260927 02:38:44.121675 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8da60306942c4204a3c8bc44eb76b7fb"
format_stamp: "Formatted at 2026-09-27 02:38:44 on dist-test-slave-zv8z"
server_key: "699312284469945900ed1e8cccf011d7"
server_key_iv: "28f49e80f19fef39f8b570d32da71572"
server_key_version: "encryptionkey@0"
I20260927 02:38:44.125311 1860 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 02:38:44.127898 1875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:44.129076 1860 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:38:44.129176 1860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-2/wal
uuid: "8da60306942c4204a3c8bc44eb76b7fb"
format_stamp: "Formatted at 2026-09-27 02:38:44 on dist-test-slave-zv8z"
server_key: "699312284469945900ed1e8cccf011d7"
server_key_iv: "28f49e80f19fef39f8b570d32da71572"
server_key_version: "encryptionkey@0"
I20260927 02:38:44.129257 1860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:38:44.141418 1860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:38:44.144035 1860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:38:44.144178 1860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:38:44.145438 1860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:38:44.145488 1860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:44.145538 1860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:38:44.145566 1860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 02:38:44 dist-test-slave-zv8z krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476724, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:38:44.155279 1860 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43353
I20260927 02:38:44.155351 1990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43353 every 8 connection(s)
I20260927 02:38:44.156333 1860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:38:44.158814 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:44.146174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36577 (local address 127.0.98.62:40591)
0927 02:38:44.146332 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:44.146336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:44.147310 (+ 974us) server_negotiation.cc:407] Connection header received
0927 02:38:44.148251 (+ 941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:44.148255 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:44.148313 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:44.148369 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:44.149958 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:44.150784 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:44.152118 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:44.152342 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:44.152429 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:44.155129 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:44.155133 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:44.155135 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:44.155164 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:44.156680 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:44.157237 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:44.157241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:44.157243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:44.157334 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:44.157665 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:44.157668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:44.157669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:44.157900 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:44.158008 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:44.158529 (+ 521us) server_negotiation.cc:299] Negotiation successful
0927 02:38:44.158650 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 02:38:44.159605 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:44.146726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40591 (local address 127.0.98.3:36577)
0927 02:38:44.147179 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:44.147217 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:44.148019 (+ 802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:44.148747 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:44.148755 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:44.149176 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:38:44.149792 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:44.149811 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:44.150913 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:44.150917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:44.152010 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:44.152020 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:44.152551 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:44.152555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:44.152698 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:44.153248 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:38:44.153267 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:38:44.154929 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:38:44.156820 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:44.156826 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:44.156839 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:44.157139 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:44.157425 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:44.157428 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:44.157429 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:44.157555 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:44.158017 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:38:44.158023 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:38:44.158293 (+ 270us) client_negotiation.cc:769] Sending connection context
0927 02:38:44.158479 (+ 186us) client_negotiation.cc:240] Negotiation successful
0927 02:38:44.158787 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":130,"threads_started":1}
I20260927 02:38:44.160684 1991 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40591
I20260927 02:38:44.160820 1991 heartbeater.cc:464] Registering TS with master...
I20260927 02:38:44.161437 1991 heartbeater.cc:511] Master 127.0.98.62:40591 requested a full tablet report, sending...
I20260927 02:38:44.162480 1506 ts_manager.cc:194] Registered new tserver with Master: 8da60306942c4204a3c8bc44eb76b7fb (127.0.98.3:43353)
I20260927 02:38:44.162760 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 1860
I20260927 02:38:44.162868 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:38:44.163177 392 external_mini_cluster.cc:1428] Setting key 43b938026e43be732ac734a6e6da3bfd
I20260927 02:38:44.163201 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36577
I20260927 02:38:44.165109 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:38:44.169879 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:44.163194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52193 (local address 127.0.98.62:40591)
0927 02:38:44.163554 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:44.163558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:44.163571 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:38:44.163604 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:44.163606 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:44.163647 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:44.163720 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:44.164261 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:44.164982 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:44.165771 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:44.165982 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:44.166060 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:44.167229 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:44.167231 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:44.167233 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:44.167254 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:44.168514 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:44.168881 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:44.168885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:44.168887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:44.168953 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:44.169184 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:44.169187 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:44.169189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:44.169338 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:44.169416 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:44.169644 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 02:38:44.169738 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277}
I20260927 02:38:44.169886 1997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:44.162904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40591 (local address 127.0.98.3:52193)
0927 02:38:44.163174 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:44.163188 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:44.163322 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:44.163830 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:44.163833 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:44.164023 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:38:44.164154 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:44.164162 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:44.165089 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:44.165092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:44.165673 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:44.165681 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:44.166071 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:44.166074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:44.166169 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:44.166710 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:38:44.166724 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:38:44.167116 (+ 392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:38:44.168612 (+ 1496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:44.168615 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:44.168617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:44.168801 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:44.169028 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:44.169031 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:44.169033 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:44.169104 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:44.169418 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:38:44.169421 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:38:44.169498 (+ 77us) client_negotiation.cc:769] Sending connection context
0927 02:38:44.169577 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 02:38:44.169710 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":87,"threads_started":1}
Sep 27 02:38:44 dist-test-slave-zv8z krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476724, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 02:38:44 dist-test-slave-zv8z krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476724, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:38:44.196558 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:44.188779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33758 (local address 127.0.98.62:40591)
0927 02:38:44.188949 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:44.188954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:44.188972 (+ 18us) server_negotiation.cc:407] Connection header received
0927 02:38:44.189035 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:44.189038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:44.189094 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:44.189195 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:44.190044 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:44.190689 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:44.191523 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:44.191737 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:44.191823 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:44.193996 (+ 2173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:44.194043 (+ 47us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:44.194045 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:44.194069 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:44.195249 (+ 1180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:44.195586 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:44.195589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:44.195590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:44.195656 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:44.195852 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:44.195854 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:44.195855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:44.196075 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:44.196168 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:44.196354 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 02:38:44.196439 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 02:38:44.199366 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33758:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:38:44.202369 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:38:44.216562 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:44.210739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41108 (local address 127.0.98.2:39007)
0927 02:38:44.212133 (+ 1394us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:44.212143 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:44.212162 (+ 19us) server_negotiation.cc:407] Connection header received
0927 02:38:44.212225 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:44.212230 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:44.212398 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:44.212527 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:38:44.212856 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:44.213592 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:44.215392 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:44.215914 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:44.216021 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:44.216188 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:44.216317 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 02:38:44.216408 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1301,"thread_start_us":102,"threads_started":1}
I20260927 02:38:44.216555 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:44.210557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39007 (local address 127.0.0.1:41108)
0927 02:38:44.211251 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:44.211265 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:44.211394 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:44.212664 (+ 1270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:44.212666 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:44.212682 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:38:44.212756 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:44.212760 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:44.213693 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:44.213696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:44.215263 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:44.215270 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:44.216030 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:44.216033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:44.216151 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:44.216193 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 02:38:44.216291 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 02:38:44.216412 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"mutex_wait_us":172,"spinlock_wait_cycles":5248,"thread_start_us":93,"threads_started":1}
I20260927 02:38:44.216964 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:44.211379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54790 (local address 127.0.98.3:43353)
0927 02:38:44.211669 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:44.211680 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:44.211890 (+ 210us) server_negotiation.cc:407] Connection header received
0927 02:38:44.212086 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:44.212144 (+ 58us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:44.212325 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:44.212462 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:38:44.212661 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:44.213760 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:44.215368 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:44.216107 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:44.216193 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:44.216343 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:44.216751 (+ 408us) server_negotiation.cc:299] Negotiation successful
0927 02:38:44.216838 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":114,"threads_started":1}
I20260927 02:38:44.217032 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:44.211293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53376 (local address 127.0.98.1:40523)
0927 02:38:44.211818 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:44.211830 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:44.211853 (+ 23us) server_negotiation.cc:407] Connection header received
0927 02:38:44.211926 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:44.211931 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:44.212147 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:44.212297 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:38:44.212578 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:44.213777 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:44.215179 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:44.215937 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:44.216645 (+ 708us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:44.216795 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:44.216837 (+ 42us) server_negotiation.cc:299] Negotiation successful
0927 02:38:44.216915 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":98,"threads_started":1}
I20260927 02:38:44.216572 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:44.210465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40523 (local address 127.0.0.1:53376)
0927 02:38:44.211213 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:44.211264 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:44.211394 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:44.212299 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:44.212305 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:44.212325 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:38:44.212457 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:44.212463 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:44.213900 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:44.213904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:44.215061 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:44.215069 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:44.216090 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:44.216094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:44.216190 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:44.216214 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 02:38:44.216270 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 02:38:44.216412 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"mutex_wait_us":21,"spinlock_wait_cycles":3712,"thread_start_us":67,"threads_started":1}
I20260927 02:38:44.217212 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:44.210777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43353 (local address 127.0.0.1:54790)
0927 02:38:44.211798 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:44.211811 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:44.211908 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:44.212452 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:44.212454 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:44.212471 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:38:44.212565 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:44.212570 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:44.213866 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:44.213869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:44.215254 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:44.215265 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:44.216397 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:44.216401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:44.216539 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:44.216560 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 02:38:44.216625 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 02:38:44.216684 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":963,"mutex_wait_us":99,"thread_start_us":78,"threads_started":1}
I20260927 02:38:44.219689 1785 tablet_service.cc:1467] Processing CreateTablet for tablet a37ad39c74a94ed3b29f995ae7ffc406 (DEFAULT_TABLE table=test-table [id=3aa8413440bc43ff8b43514c00cca165]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:38:44.219689 1925 tablet_service.cc:1467] Processing CreateTablet for tablet a37ad39c74a94ed3b29f995ae7ffc406 (DEFAULT_TABLE table=test-table [id=3aa8413440bc43ff8b43514c00cca165]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:38:44.220587 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a37ad39c74a94ed3b29f995ae7ffc406. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:38:44.220607 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a37ad39c74a94ed3b29f995ae7ffc406. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:38:44.227934 2014 tablet_bootstrap.cc:492] T a37ad39c74a94ed3b29f995ae7ffc406 P 428bd355e94941368a1b1f8c964d5bac: Bootstrap starting.
I20260927 02:38:44.227540 1645 tablet_service.cc:1467] Processing CreateTablet for tablet a37ad39c74a94ed3b29f995ae7ffc406 (DEFAULT_TABLE table=test-table [id=3aa8413440bc43ff8b43514c00cca165]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:38:44.227945 2015 tablet_bootstrap.cc:492] T a37ad39c74a94ed3b29f995ae7ffc406 P 8da60306942c4204a3c8bc44eb76b7fb: Bootstrap starting.
I20260927 02:38:44.228556 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a37ad39c74a94ed3b29f995ae7ffc406. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:38:44.229696 2014 tablet_bootstrap.cc:654] T a37ad39c74a94ed3b29f995ae7ffc406 P 428bd355e94941368a1b1f8c964d5bac: Neither blocks nor log segments found. Creating new log.
I20260927 02:38:44.230149 2015 tablet_bootstrap.cc:654] T a37ad39c74a94ed3b29f995ae7ffc406 P 8da60306942c4204a3c8bc44eb76b7fb: Neither blocks nor log segments found. Creating new log.
I20260927 02:38:44.230400 2014 log.cc:824] T a37ad39c74a94ed3b29f995ae7ffc406 P 428bd355e94941368a1b1f8c964d5bac: Log is configured to *not* fsync() on all Append() calls
I20260927 02:38:44.230940 2015 log.cc:824] T a37ad39c74a94ed3b29f995ae7ffc406 P 8da60306942c4204a3c8bc44eb76b7fb: Log is configured to *not* fsync() on all Append() calls
I20260927 02:38:44.232188 2014 tablet_bootstrap.cc:492] T a37ad39c74a94ed3b29f995ae7ffc406 P 428bd355e94941368a1b1f8c964d5bac: No bootstrap required, opened a new log
I20260927 02:38:44.232368 2014 ts_tablet_manager.cc:1397] T a37ad39c74a94ed3b29f995ae7ffc406 P 428bd355e94941368a1b1f8c964d5bac: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 02:38:44.232880 2015 tablet_bootstrap.cc:492] T a37ad39c74a94ed3b29f995ae7ffc406 P 8da60306942c4204a3c8bc44eb76b7fb: No bootstrap required, opened a new log
I20260927 02:38:44.233047 2015 ts_tablet_manager.cc:1397] T a37ad39c74a94ed3b29f995ae7ffc406 P 8da60306942c4204a3c8bc44eb76b7fb: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 02:38:44.235337 2014 raft_consensus.cc:348] T a37ad39c74a94ed3b29f995ae7ffc406 P 428bd355e94941368a1b1f8c964d5bac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f8e9f42b7c4601944a51a87c1cf4cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40523 } } peers { permanent_uuid: "428bd355e94941368a1b1f8c964d5bac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39007 } } peers { permanent_uuid: "8da60306942c4204a3c8bc44eb76b7fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43353 } }
I20260927 02:38:44.235536 2014 raft_consensus.cc:374] T a37ad39c74a94ed3b29f995ae7ffc406 P 428bd355e94941368a1b1f8c964d5bac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:38:44.235589 2014 raft_consensus.cc:729] T a37ad39c74a94ed3b29f995ae7ffc406 P 428bd355e94941368a1b1f8c964d5bac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428bd355e94941368a1b1f8c964d5bac, State: Initialized, Role: FOLLOWER
I20260927 02:38:44.235997 2014 consensus_queue.cc:260] T a37ad39c74a94ed3b29f995ae7ffc406 P 428bd355e94941368a1b1f8c964d5bac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f8e9f42b7c4601944a51a87c1cf4cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40523 } } peers { permanent_uuid: "428bd355e94941368a1b1f8c964d5bac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39007 } } peers { permanent_uuid: "8da60306942c4204a3c8bc44eb76b7fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43353 } }
I20260927 02:38:44.236302 2018 tablet_bootstrap.cc:492] T a37ad39c74a94ed3b29f995ae7ffc406 P 90f8e9f42b7c4601944a51a87c1cf4cd: Bootstrap starting.
I20260927 02:38:44.236948 2014 ts_tablet_manager.cc:1428] T a37ad39c74a94ed3b29f995ae7ffc406 P 428bd355e94941368a1b1f8c964d5bac: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 02:38:44.236796 2015 raft_consensus.cc:348] T a37ad39c74a94ed3b29f995ae7ffc406 P 8da60306942c4204a3c8bc44eb76b7fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f8e9f42b7c4601944a51a87c1cf4cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40523 } } peers { permanent_uuid: "428bd355e94941368a1b1f8c964d5bac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39007 } } peers { permanent_uuid: "8da60306942c4204a3c8bc44eb76b7fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43353 } }
I20260927 02:38:44.237025 2015 raft_consensus.cc:374] T a37ad39c74a94ed3b29f995ae7ffc406 P 8da60306942c4204a3c8bc44eb76b7fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:38:44.237085 2015 raft_consensus.cc:729] T a37ad39c74a94ed3b29f995ae7ffc406 P 8da60306942c4204a3c8bc44eb76b7fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8da60306942c4204a3c8bc44eb76b7fb, State: Initialized, Role: FOLLOWER
I20260927 02:38:44.237277 1851 heartbeater.cc:503] Master 127.0.98.62:40591 was elected leader, sending a full tablet report...
I20260927 02:38:44.237519 2015 consensus_queue.cc:260] T a37ad39c74a94ed3b29f995ae7ffc406 P 8da60306942c4204a3c8bc44eb76b7fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f8e9f42b7c4601944a51a87c1cf4cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40523 } } peers { permanent_uuid: "428bd355e94941368a1b1f8c964d5bac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39007 } } peers { permanent_uuid: "8da60306942c4204a3c8bc44eb76b7fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43353 } }
I20260927 02:38:44.238427 2018 tablet_bootstrap.cc:654] T a37ad39c74a94ed3b29f995ae7ffc406 P 90f8e9f42b7c4601944a51a87c1cf4cd: Neither blocks nor log segments found. Creating new log.
I20260927 02:38:44.238451 2015 ts_tablet_manager.cc:1428] T a37ad39c74a94ed3b29f995ae7ffc406 P 8da60306942c4204a3c8bc44eb76b7fb: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 02:38:44.238490 1991 heartbeater.cc:503] Master 127.0.98.62:40591 was elected leader, sending a full tablet report...
I20260927 02:38:44.239233 2018 log.cc:824] T a37ad39c74a94ed3b29f995ae7ffc406 P 90f8e9f42b7c4601944a51a87c1cf4cd: Log is configured to *not* fsync() on all Append() calls
I20260927 02:38:44.241082 2018 tablet_bootstrap.cc:492] T a37ad39c74a94ed3b29f995ae7ffc406 P 90f8e9f42b7c4601944a51a87c1cf4cd: No bootstrap required, opened a new log
I20260927 02:38:44.241256 2018 ts_tablet_manager.cc:1397] T a37ad39c74a94ed3b29f995ae7ffc406 P 90f8e9f42b7c4601944a51a87c1cf4cd: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 02:38:44.244042 2018 raft_consensus.cc:348] T a37ad39c74a94ed3b29f995ae7ffc406 P 90f8e9f42b7c4601944a51a87c1cf4cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f8e9f42b7c4601944a51a87c1cf4cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40523 } } peers { permanent_uuid: "428bd355e94941368a1b1f8c964d5bac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39007 } } peers { permanent_uuid: "8da60306942c4204a3c8bc44eb76b7fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43353 } }
I20260927 02:38:44.244235 2018 raft_consensus.cc:374] T a37ad39c74a94ed3b29f995ae7ffc406 P 90f8e9f42b7c4601944a51a87c1cf4cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:38:44.244266 2018 raft_consensus.cc:729] T a37ad39c74a94ed3b29f995ae7ffc406 P 90f8e9f42b7c4601944a51a87c1cf4cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90f8e9f42b7c4601944a51a87c1cf4cd, State: Initialized, Role: FOLLOWER
I20260927 02:38:44.244580 2018 consensus_queue.cc:260] T a37ad39c74a94ed3b29f995ae7ffc406 P 90f8e9f42b7c4601944a51a87c1cf4cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f8e9f42b7c4601944a51a87c1cf4cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40523 } } peers { permanent_uuid: "428bd355e94941368a1b1f8c964d5bac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39007 } } peers { permanent_uuid: "8da60306942c4204a3c8bc44eb76b7fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43353 } }
I20260927 02:38:44.245133 1711 heartbeater.cc:503] Master 127.0.98.62:40591 was elected leader, sending a full tablet report...
I20260927 02:38:44.245314 2018 ts_tablet_manager.cc:1428] T a37ad39c74a94ed3b29f995ae7ffc406 P 90f8e9f42b7c4601944a51a87c1cf4cd: Time spent starting tablet: real 0.004s user 0.001s sys 0.004s
W20260927 02:38:44.249363 1712 tablet.cc:2369] T a37ad39c74a94ed3b29f995ae7ffc406 P 90f8e9f42b7c4601944a51a87c1cf4cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:38:44.276224 2019 raft_consensus.cc:482] T a37ad39c74a94ed3b29f995ae7ffc406 P 428bd355e94941368a1b1f8c964d5bac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:38:44.276405 2019 raft_consensus.cc:504] T a37ad39c74a94ed3b29f995ae7ffc406 P 428bd355e94941368a1b1f8c964d5bac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f8e9f42b7c4601944a51a87c1cf4cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40523 } } peers { permanent_uuid: "428bd355e94941368a1b1f8c964d5bac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39007 } } peers { permanent_uuid: "8da60306942c4204a3c8bc44eb76b7fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43353 } }
I20260927 02:38:44.277450 2019 leader_election.cc:288] T a37ad39c74a94ed3b29f995ae7ffc406 P 428bd355e94941368a1b1f8c964d5bac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90f8e9f42b7c4601944a51a87c1cf4cd (127.0.98.1:40523), 8da60306942c4204a3c8bc44eb76b7fb (127.0.98.3:43353)
I20260927 02:38:44.281306 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:44.277745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51615 (local address 127.0.98.1:40523)
0927 02:38:44.277870 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:44.277874 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:44.277895 (+ 21us) server_negotiation.cc:407] Connection header received
0927 02:38:44.278054 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:44.278058 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:44.278111 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:44.278182 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:38:44.278534 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:44.279085 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:44.280169 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:44.280777 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:44.280851 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:44.280900 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:44.281154 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 02:38:44.281207 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 02:38:44.281339 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:44.277666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40523 (local address 127.0.98.2:51615)
0927 02:38:44.277840 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:44.277855 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:44.277941 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:44.278226 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:44.278229 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:44.278279 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:38:44.278401 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:44.278408 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:44.279190 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:44.279193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:44.280048 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:44.280057 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:44.280874 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:44.280878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:44.280983 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:44.281008 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 02:38:44.281143 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 02:38:44.281214 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20260927 02:38:44.281807 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:44.277722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43353 (local address 127.0.98.2:60801)
0927 02:38:44.278083 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:44.278095 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:44.278193 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:44.278422 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:44.278424 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:44.278437 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:38:44.278565 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:44.278571 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:44.279265 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:44.279268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:44.280567 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:44.280577 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:44.281396 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:44.281400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:44.281494 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:44.281516 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 02:38:44.281617 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 02:38:44.281677 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":124,"threads_started":1}
I20260927 02:38:44.281822 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:44.277801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60801 (local address 127.0.98.3:43353)
0927 02:38:44.277900 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:44.277904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:44.278201 (+ 297us) server_negotiation.cc:407] Connection header received
0927 02:38:44.278283 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:44.278285 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:44.278333 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:44.278415 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:38:44.278666 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:44.279165 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:44.280730 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:44.281267 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:44.281356 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:44.281411 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:44.281652 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 02:38:44.281727 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 02:38:44.282014 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a37ad39c74a94ed3b29f995ae7ffc406" candidate_uuid: "428bd355e94941368a1b1f8c964d5bac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90f8e9f42b7c4601944a51a87c1cf4cd" is_pre_election: true
I20260927 02:38:44.282356 1665 raft_consensus.cc:2454] T a37ad39c74a94ed3b29f995ae7ffc406 P 90f8e9f42b7c4601944a51a87c1cf4cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428bd355e94941368a1b1f8c964d5bac in term 0.
I20260927 02:38:44.282424 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a37ad39c74a94ed3b29f995ae7ffc406" candidate_uuid: "428bd355e94941368a1b1f8c964d5bac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8da60306942c4204a3c8bc44eb76b7fb" is_pre_election: true
I20260927 02:38:44.282691 1945 raft_consensus.cc:2454] T a37ad39c74a94ed3b29f995ae7ffc406 P 8da60306942c4204a3c8bc44eb76b7fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428bd355e94941368a1b1f8c964d5bac in term 0.
I20260927 02:38:44.282804 1737 leader_election.cc:302] T a37ad39c74a94ed3b29f995ae7ffc406 P 428bd355e94941368a1b1f8c964d5bac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428bd355e94941368a1b1f8c964d5bac, 90f8e9f42b7c4601944a51a87c1cf4cd; no voters:
I20260927 02:38:44.283067 2019 raft_consensus.cc:2781] T a37ad39c74a94ed3b29f995ae7ffc406 P 428bd355e94941368a1b1f8c964d5bac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:38:44.283169 2019 raft_consensus.cc:482] T a37ad39c74a94ed3b29f995ae7ffc406 P 428bd355e94941368a1b1f8c964d5bac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:38:44.283222 2019 raft_consensus.cc:3037] T a37ad39c74a94ed3b29f995ae7ffc406 P 428bd355e94941368a1b1f8c964d5bac [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:38:44.284286 2019 raft_consensus.cc:504] T a37ad39c74a94ed3b29f995ae7ffc406 P 428bd355e94941368a1b1f8c964d5bac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f8e9f42b7c4601944a51a87c1cf4cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40523 } } peers { permanent_uuid: "428bd355e94941368a1b1f8c964d5bac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39007 } } peers { permanent_uuid: "8da60306942c4204a3c8bc44eb76b7fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43353 } }
I20260927 02:38:44.284576 2019 leader_election.cc:288] T a37ad39c74a94ed3b29f995ae7ffc406 P 428bd355e94941368a1b1f8c964d5bac [CANDIDATE]: Term 1 election: Requested vote from peers 90f8e9f42b7c4601944a51a87c1cf4cd (127.0.98.1:40523), 8da60306942c4204a3c8bc44eb76b7fb (127.0.98.3:43353)
I20260927 02:38:44.284909 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a37ad39c74a94ed3b29f995ae7ffc406" candidate_uuid: "428bd355e94941368a1b1f8c964d5bac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90f8e9f42b7c4601944a51a87c1cf4cd"
I20260927 02:38:44.284909 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a37ad39c74a94ed3b29f995ae7ffc406" candidate_uuid: "428bd355e94941368a1b1f8c964d5bac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8da60306942c4204a3c8bc44eb76b7fb"
I20260927 02:38:44.285027 1665 raft_consensus.cc:3037] T a37ad39c74a94ed3b29f995ae7ffc406 P 90f8e9f42b7c4601944a51a87c1cf4cd [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:38:44.285027 1945 raft_consensus.cc:3037] T a37ad39c74a94ed3b29f995ae7ffc406 P 8da60306942c4204a3c8bc44eb76b7fb [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:38:44.286201 1945 raft_consensus.cc:2454] T a37ad39c74a94ed3b29f995ae7ffc406 P 8da60306942c4204a3c8bc44eb76b7fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428bd355e94941368a1b1f8c964d5bac in term 1.
I20260927 02:38:44.286201 1665 raft_consensus.cc:2454] T a37ad39c74a94ed3b29f995ae7ffc406 P 90f8e9f42b7c4601944a51a87c1cf4cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428bd355e94941368a1b1f8c964d5bac in term 1.
I20260927 02:38:44.286507 1737 leader_election.cc:302] T a37ad39c74a94ed3b29f995ae7ffc406 P 428bd355e94941368a1b1f8c964d5bac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428bd355e94941368a1b1f8c964d5bac, 90f8e9f42b7c4601944a51a87c1cf4cd; no voters:
I20260927 02:38:44.286685 2019 raft_consensus.cc:2781] T a37ad39c74a94ed3b29f995ae7ffc406 P 428bd355e94941368a1b1f8c964d5bac [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:38:44.286891 2019 raft_consensus.cc:686] T a37ad39c74a94ed3b29f995ae7ffc406 P 428bd355e94941368a1b1f8c964d5bac [term 1 LEADER]: Becoming Leader. State: Replica: 428bd355e94941368a1b1f8c964d5bac, State: Running, Role: LEADER
I20260927 02:38:44.287214 2019 consensus_queue.cc:237] T a37ad39c74a94ed3b29f995ae7ffc406 P 428bd355e94941368a1b1f8c964d5bac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f8e9f42b7c4601944a51a87c1cf4cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40523 } } peers { permanent_uuid: "428bd355e94941368a1b1f8c964d5bac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39007 } } peers { permanent_uuid: "8da60306942c4204a3c8bc44eb76b7fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43353 } }
I20260927 02:38:44.290056 1506 catalog_manager.cc:5495] T a37ad39c74a94ed3b29f995ae7ffc406 P 428bd355e94941368a1b1f8c964d5bac reported cstate change: term changed from 0 to 1, leader changed from <none> to 428bd355e94941368a1b1f8c964d5bac (127.0.98.2). New cstate: current_term: 1 leader_uuid: "428bd355e94941368a1b1f8c964d5bac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f8e9f42b7c4601944a51a87c1cf4cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "428bd355e94941368a1b1f8c964d5bac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8da60306942c4204a3c8bc44eb76b7fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43353 } health_report { overall_health: UNKNOWN } } }
I20260927 02:38:44.318307 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:44.314931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41110 (local address 127.0.98.2:39007)
0927 02:38:44.315083 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:44.315086 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:44.315100 (+ 14us) server_negotiation.cc:407] Connection header received
0927 02:38:44.315228 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:44.315230 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:44.315282 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:44.315365 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:38:44.315533 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:44.316038 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:44.316950 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:44.317172 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:44.317255 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:44.317496 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:38:44.317967 (+ 471us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:38:44.318060 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:44.318123 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 02:38:44.318173 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 02:38:44.324057 1665 raft_consensus.cc:1261] T a37ad39c74a94ed3b29f995ae7ffc406 P 90f8e9f42b7c4601944a51a87c1cf4cd [term 1 FOLLOWER]: Refusing update from remote peer 428bd355e94941368a1b1f8c964d5bac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:38:44.324363 1945 raft_consensus.cc:1261] T a37ad39c74a94ed3b29f995ae7ffc406 P 8da60306942c4204a3c8bc44eb76b7fb [term 1 FOLLOWER]: Refusing update from remote peer 428bd355e94941368a1b1f8c964d5bac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:38:44.324677 2024 consensus_queue.cc:1035] T a37ad39c74a94ed3b29f995ae7ffc406 P 428bd355e94941368a1b1f8c964d5bac [LEADER]: Connected to new peer: Peer: permanent_uuid: "90f8e9f42b7c4601944a51a87c1cf4cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:38:44.324927 2019 consensus_queue.cc:1035] T a37ad39c74a94ed3b29f995ae7ffc406 P 428bd355e94941368a1b1f8c964d5bac [LEADER]: Connected to new peer: Peer: permanent_uuid: "8da60306942c4204a3c8bc44eb76b7fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:38:44.344254 1506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:38:44.344444 1506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:38:44.346889 1506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P edff1a4c13434b2e9d0ddfc6409df266: Sending DeleteTablet for 3 replicas of tablet a37ad39c74a94ed3b29f995ae7ffc406
I20260927 02:38:44.347674 1785 tablet_service.cc:1514] Processing DeleteTablet for tablet a37ad39c74a94ed3b29f995ae7ffc406 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:38:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41108
I20260927 02:38:44.347704 1645 tablet_service.cc:1514] Processing DeleteTablet for tablet a37ad39c74a94ed3b29f995ae7ffc406 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:38:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53376
I20260927 02:38:44.348225 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet a37ad39c74a94ed3b29f995ae7ffc406 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:38:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54790
I20260927 02:38:44.348666 2038 tablet_replica.cc:330] T a37ad39c74a94ed3b29f995ae7ffc406 P 428bd355e94941368a1b1f8c964d5bac: stopping tablet replica
I20260927 02:38:44.348695 2039 tablet_replica.cc:330] T a37ad39c74a94ed3b29f995ae7ffc406 P 8da60306942c4204a3c8bc44eb76b7fb: stopping tablet replica
I20260927 02:38:44.348963 2038 raft_consensus.cc:2229] T a37ad39c74a94ed3b29f995ae7ffc406 P 428bd355e94941368a1b1f8c964d5bac [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:38:44.348973 2039 raft_consensus.cc:2229] T a37ad39c74a94ed3b29f995ae7ffc406 P 8da60306942c4204a3c8bc44eb76b7fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:38:44.348874 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33758:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 02:38:44.349289 2039 raft_consensus.cc:2258] T a37ad39c74a94ed3b29f995ae7ffc406 P 8da60306942c4204a3c8bc44eb76b7fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:38:44.349334 2038 raft_consensus.cc:2258] T a37ad39c74a94ed3b29f995ae7ffc406 P 428bd355e94941368a1b1f8c964d5bac [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 02:38:44.349344 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:38:44.349663 2037 tablet_replica.cc:330] T a37ad39c74a94ed3b29f995ae7ffc406 P 90f8e9f42b7c4601944a51a87c1cf4cd: stopping tablet replica
I20260927 02:38:44.349951 2037 raft_consensus.cc:2229] T a37ad39c74a94ed3b29f995ae7ffc406 P 90f8e9f42b7c4601944a51a87c1cf4cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:38:44.350229 2037 raft_consensus.cc:2258] T a37ad39c74a94ed3b29f995ae7ffc406 P 90f8e9f42b7c4601944a51a87c1cf4cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:38:44.351374 2039 ts_tablet_manager.cc:1905] T a37ad39c74a94ed3b29f995ae7ffc406 P 8da60306942c4204a3c8bc44eb76b7fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:38:44.351549 2037 ts_tablet_manager.cc:1905] T a37ad39c74a94ed3b29f995ae7ffc406 P 90f8e9f42b7c4601944a51a87c1cf4cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:38:44.353127 2038 ts_tablet_manager.cc:1905] T a37ad39c74a94ed3b29f995ae7ffc406 P 428bd355e94941368a1b1f8c964d5bac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:38:44.354166 2037 ts_tablet_manager.cc:1918] T a37ad39c74a94ed3b29f995ae7ffc406 P 90f8e9f42b7c4601944a51a87c1cf4cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:38:44.354283 2037 log.cc:1196] T a37ad39c74a94ed3b29f995ae7ffc406 P 90f8e9f42b7c4601944a51a87c1cf4cd: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-0/wal/wals/a37ad39c74a94ed3b29f995ae7ffc406
I20260927 02:38:44.354219 2039 ts_tablet_manager.cc:1918] T a37ad39c74a94ed3b29f995ae7ffc406 P 8da60306942c4204a3c8bc44eb76b7fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:38:44.354499 2039 log.cc:1196] T a37ad39c74a94ed3b29f995ae7ffc406 P 8da60306942c4204a3c8bc44eb76b7fb: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-2/wal/wals/a37ad39c74a94ed3b29f995ae7ffc406
I20260927 02:38:44.354573 2037 ts_tablet_manager.cc:1939] T a37ad39c74a94ed3b29f995ae7ffc406 P 90f8e9f42b7c4601944a51a87c1cf4cd: Deleting consensus metadata
I20260927 02:38:44.354923 2039 ts_tablet_manager.cc:1939] T a37ad39c74a94ed3b29f995ae7ffc406 P 8da60306942c4204a3c8bc44eb76b7fb: Deleting consensus metadata
I20260927 02:38:44.355392 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 17d4e7de7b6f43cbafd2706d19a04cb5 (DEFAULT_TABLE table=test-table [id=745c5d073dd445189dd4dd2976cf705d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:38:44.355574 2038 ts_tablet_manager.cc:1918] T a37ad39c74a94ed3b29f995ae7ffc406 P 428bd355e94941368a1b1f8c964d5bac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:38:44.355648 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17d4e7de7b6f43cbafd2706d19a04cb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:38:44.355679 2038 log.cc:1196] T a37ad39c74a94ed3b29f995ae7ffc406 P 428bd355e94941368a1b1f8c964d5bac: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-1/wal/wals/a37ad39c74a94ed3b29f995ae7ffc406
I20260927 02:38:44.355712 1493 catalog_manager.cc:4841] TS 90f8e9f42b7c4601944a51a87c1cf4cd (127.0.98.1:40523): tablet a37ad39c74a94ed3b29f995ae7ffc406 (table test-table [id=3aa8413440bc43ff8b43514c00cca165]) successfully deleted
I20260927 02:38:44.355962 2038 ts_tablet_manager.cc:1939] T a37ad39c74a94ed3b29f995ae7ffc406 P 428bd355e94941368a1b1f8c964d5bac: Deleting consensus metadata
I20260927 02:38:44.356704 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 17d4e7de7b6f43cbafd2706d19a04cb5 (DEFAULT_TABLE table=test-table [id=745c5d073dd445189dd4dd2976cf705d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:38:44.356921 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17d4e7de7b6f43cbafd2706d19a04cb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:38:44.357499 1494 catalog_manager.cc:4841] TS 8da60306942c4204a3c8bc44eb76b7fb (127.0.98.3:43353): tablet a37ad39c74a94ed3b29f995ae7ffc406 (table test-table [id=3aa8413440bc43ff8b43514c00cca165]) successfully deleted
I20260927 02:38:44.357784 2018 tablet_bootstrap.cc:492] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 90f8e9f42b7c4601944a51a87c1cf4cd: Bootstrap starting.
I20260927 02:38:44.357944 1495 catalog_manager.cc:4841] TS 428bd355e94941368a1b1f8c964d5bac (127.0.98.2:39007): tablet a37ad39c74a94ed3b29f995ae7ffc406 (table test-table [id=3aa8413440bc43ff8b43514c00cca165]) successfully deleted
I20260927 02:38:44.358724 2018 tablet_bootstrap.cc:654] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 90f8e9f42b7c4601944a51a87c1cf4cd: Neither blocks nor log segments found. Creating new log.
I20260927 02:38:44.358769 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 17d4e7de7b6f43cbafd2706d19a04cb5 (DEFAULT_TABLE table=test-table [id=745c5d073dd445189dd4dd2976cf705d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:38:44.358980 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17d4e7de7b6f43cbafd2706d19a04cb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:38:44.359129 2015 tablet_bootstrap.cc:492] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 8da60306942c4204a3c8bc44eb76b7fb: Bootstrap starting.
I20260927 02:38:44.359860 2018 tablet_bootstrap.cc:492] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 90f8e9f42b7c4601944a51a87c1cf4cd: No bootstrap required, opened a new log
I20260927 02:38:44.359933 2018 ts_tablet_manager.cc:1397] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 90f8e9f42b7c4601944a51a87c1cf4cd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260927 02:38:44.359992 2015 tablet_bootstrap.cc:654] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 8da60306942c4204a3c8bc44eb76b7fb: Neither blocks nor log segments found. Creating new log.
I20260927 02:38:44.360658 2018 raft_consensus.cc:348] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 90f8e9f42b7c4601944a51a87c1cf4cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f8e9f42b7c4601944a51a87c1cf4cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40523 } } peers { permanent_uuid: "428bd355e94941368a1b1f8c964d5bac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39007 } } peers { permanent_uuid: "8da60306942c4204a3c8bc44eb76b7fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43353 } }
I20260927 02:38:44.360762 2018 raft_consensus.cc:374] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 90f8e9f42b7c4601944a51a87c1cf4cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:38:44.360793 2018 raft_consensus.cc:729] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 90f8e9f42b7c4601944a51a87c1cf4cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90f8e9f42b7c4601944a51a87c1cf4cd, State: Initialized, Role: FOLLOWER
I20260927 02:38:44.360906 2018 consensus_queue.cc:260] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 90f8e9f42b7c4601944a51a87c1cf4cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f8e9f42b7c4601944a51a87c1cf4cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40523 } } peers { permanent_uuid: "428bd355e94941368a1b1f8c964d5bac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39007 } } peers { permanent_uuid: "8da60306942c4204a3c8bc44eb76b7fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43353 } }
I20260927 02:38:44.361086 2015 tablet_bootstrap.cc:492] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 8da60306942c4204a3c8bc44eb76b7fb: No bootstrap required, opened a new log
I20260927 02:38:44.361164 2015 ts_tablet_manager.cc:1397] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 8da60306942c4204a3c8bc44eb76b7fb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:38:44.361186 2018 ts_tablet_manager.cc:1428] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 90f8e9f42b7c4601944a51a87c1cf4cd: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260927 02:38:44.361243 2014 tablet_bootstrap.cc:492] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 428bd355e94941368a1b1f8c964d5bac: Bootstrap starting.
I20260927 02:38:44.361559 2015 raft_consensus.cc:348] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 8da60306942c4204a3c8bc44eb76b7fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f8e9f42b7c4601944a51a87c1cf4cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40523 } } peers { permanent_uuid: "428bd355e94941368a1b1f8c964d5bac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39007 } } peers { permanent_uuid: "8da60306942c4204a3c8bc44eb76b7fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43353 } }
I20260927 02:38:44.361660 2015 raft_consensus.cc:374] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 8da60306942c4204a3c8bc44eb76b7fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:38:44.361688 2015 raft_consensus.cc:729] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 8da60306942c4204a3c8bc44eb76b7fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8da60306942c4204a3c8bc44eb76b7fb, State: Initialized, Role: FOLLOWER
I20260927 02:38:44.361802 2015 consensus_queue.cc:260] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 8da60306942c4204a3c8bc44eb76b7fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f8e9f42b7c4601944a51a87c1cf4cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40523 } } peers { permanent_uuid: "428bd355e94941368a1b1f8c964d5bac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39007 } } peers { permanent_uuid: "8da60306942c4204a3c8bc44eb76b7fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43353 } }
I20260927 02:38:44.362033 2015 ts_tablet_manager.cc:1428] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 8da60306942c4204a3c8bc44eb76b7fb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 02:38:44.362103 2014 tablet_bootstrap.cc:654] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 428bd355e94941368a1b1f8c964d5bac: Neither blocks nor log segments found. Creating new log.
I20260927 02:38:44.364144 2014 tablet_bootstrap.cc:492] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 428bd355e94941368a1b1f8c964d5bac: No bootstrap required, opened a new log
I20260927 02:38:44.364223 2014 ts_tablet_manager.cc:1397] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 428bd355e94941368a1b1f8c964d5bac: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 02:38:44.364611 2014 raft_consensus.cc:348] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 428bd355e94941368a1b1f8c964d5bac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f8e9f42b7c4601944a51a87c1cf4cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40523 } } peers { permanent_uuid: "428bd355e94941368a1b1f8c964d5bac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39007 } } peers { permanent_uuid: "8da60306942c4204a3c8bc44eb76b7fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43353 } }
I20260927 02:38:44.364718 2014 raft_consensus.cc:374] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 428bd355e94941368a1b1f8c964d5bac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:38:44.364751 2014 raft_consensus.cc:729] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 428bd355e94941368a1b1f8c964d5bac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428bd355e94941368a1b1f8c964d5bac, State: Initialized, Role: FOLLOWER
I20260927 02:38:44.364852 2014 consensus_queue.cc:260] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 428bd355e94941368a1b1f8c964d5bac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f8e9f42b7c4601944a51a87c1cf4cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40523 } } peers { permanent_uuid: "428bd355e94941368a1b1f8c964d5bac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39007 } } peers { permanent_uuid: "8da60306942c4204a3c8bc44eb76b7fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43353 } }
I20260927 02:38:44.365079 2014 ts_tablet_manager.cc:1428] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 428bd355e94941368a1b1f8c964d5bac: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260927 02:38:44.407747 1992 tablet.cc:2369] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 8da60306942c4204a3c8bc44eb76b7fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:38:44.455480 2032 raft_consensus.cc:482] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 428bd355e94941368a1b1f8c964d5bac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:38:44.455617 2032 raft_consensus.cc:504] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 428bd355e94941368a1b1f8c964d5bac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f8e9f42b7c4601944a51a87c1cf4cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40523 } } peers { permanent_uuid: "428bd355e94941368a1b1f8c964d5bac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39007 } } peers { permanent_uuid: "8da60306942c4204a3c8bc44eb76b7fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43353 } }
I20260927 02:38:44.456120 2032 leader_election.cc:288] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 428bd355e94941368a1b1f8c964d5bac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90f8e9f42b7c4601944a51a87c1cf4cd (127.0.98.1:40523), 8da60306942c4204a3c8bc44eb76b7fb (127.0.98.3:43353)
I20260927 02:38:44.456574 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17d4e7de7b6f43cbafd2706d19a04cb5" candidate_uuid: "428bd355e94941368a1b1f8c964d5bac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8da60306942c4204a3c8bc44eb76b7fb" is_pre_election: true
I20260927 02:38:44.456574 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17d4e7de7b6f43cbafd2706d19a04cb5" candidate_uuid: "428bd355e94941368a1b1f8c964d5bac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90f8e9f42b7c4601944a51a87c1cf4cd" is_pre_election: true
I20260927 02:38:44.456722 1945 raft_consensus.cc:2454] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 8da60306942c4204a3c8bc44eb76b7fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428bd355e94941368a1b1f8c964d5bac in term 0.
I20260927 02:38:44.456722 1665 raft_consensus.cc:2454] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 90f8e9f42b7c4601944a51a87c1cf4cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428bd355e94941368a1b1f8c964d5bac in term 0.
I20260927 02:38:44.457036 1738 leader_election.cc:302] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 428bd355e94941368a1b1f8c964d5bac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428bd355e94941368a1b1f8c964d5bac, 8da60306942c4204a3c8bc44eb76b7fb; no voters:
I20260927 02:38:44.457239 2032 raft_consensus.cc:2781] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 428bd355e94941368a1b1f8c964d5bac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:38:44.457300 2032 raft_consensus.cc:482] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 428bd355e94941368a1b1f8c964d5bac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:38:44.457329 2032 raft_consensus.cc:3037] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 428bd355e94941368a1b1f8c964d5bac [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:38:44.458462 2032 raft_consensus.cc:504] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 428bd355e94941368a1b1f8c964d5bac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f8e9f42b7c4601944a51a87c1cf4cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40523 } } peers { permanent_uuid: "428bd355e94941368a1b1f8c964d5bac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39007 } } peers { permanent_uuid: "8da60306942c4204a3c8bc44eb76b7fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43353 } }
I20260927 02:38:44.458804 2032 leader_election.cc:288] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 428bd355e94941368a1b1f8c964d5bac [CANDIDATE]: Term 1 election: Requested vote from peers 90f8e9f42b7c4601944a51a87c1cf4cd (127.0.98.1:40523), 8da60306942c4204a3c8bc44eb76b7fb (127.0.98.3:43353)
I20260927 02:38:44.459161 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17d4e7de7b6f43cbafd2706d19a04cb5" candidate_uuid: "428bd355e94941368a1b1f8c964d5bac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90f8e9f42b7c4601944a51a87c1cf4cd"
I20260927 02:38:44.459221 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17d4e7de7b6f43cbafd2706d19a04cb5" candidate_uuid: "428bd355e94941368a1b1f8c964d5bac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8da60306942c4204a3c8bc44eb76b7fb"
I20260927 02:38:44.459285 1665 raft_consensus.cc:3037] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 90f8e9f42b7c4601944a51a87c1cf4cd [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:38:44.459357 1945 raft_consensus.cc:3037] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 8da60306942c4204a3c8bc44eb76b7fb [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:38:44.460135 1665 raft_consensus.cc:2454] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 90f8e9f42b7c4601944a51a87c1cf4cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428bd355e94941368a1b1f8c964d5bac in term 1.
I20260927 02:38:44.460384 1737 leader_election.cc:302] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 428bd355e94941368a1b1f8c964d5bac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428bd355e94941368a1b1f8c964d5bac, 90f8e9f42b7c4601944a51a87c1cf4cd; no voters:
I20260927 02:38:44.460500 1945 raft_consensus.cc:2454] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 8da60306942c4204a3c8bc44eb76b7fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428bd355e94941368a1b1f8c964d5bac in term 1.
I20260927 02:38:44.460538 2032 raft_consensus.cc:2781] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 428bd355e94941368a1b1f8c964d5bac [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:38:44.460608 2032 raft_consensus.cc:686] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 428bd355e94941368a1b1f8c964d5bac [term 1 LEADER]: Becoming Leader. State: Replica: 428bd355e94941368a1b1f8c964d5bac, State: Running, Role: LEADER
I20260927 02:38:44.460747 2032 consensus_queue.cc:237] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 428bd355e94941368a1b1f8c964d5bac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f8e9f42b7c4601944a51a87c1cf4cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40523 } } peers { permanent_uuid: "428bd355e94941368a1b1f8c964d5bac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39007 } } peers { permanent_uuid: "8da60306942c4204a3c8bc44eb76b7fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43353 } }
W20260927 02:38:44.461483 1852 tablet.cc:2369] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 428bd355e94941368a1b1f8c964d5bac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:38:44.462443 1508 catalog_manager.cc:5495] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 428bd355e94941368a1b1f8c964d5bac reported cstate change: term changed from 0 to 1, leader changed from <none> to 428bd355e94941368a1b1f8c964d5bac (127.0.98.2). New cstate: current_term: 1 leader_uuid: "428bd355e94941368a1b1f8c964d5bac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f8e9f42b7c4601944a51a87c1cf4cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "428bd355e94941368a1b1f8c964d5bac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8da60306942c4204a3c8bc44eb76b7fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43353 } health_report { overall_health: UNKNOWN } } }
Sep 27 02:38:44 dist-test-slave-zv8z krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476723, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:38:44.511914 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:44.503945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40591 (local address 127.0.0.1:33760)
0927 02:38:44.504285 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:44.504299 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:44.504404 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:44.504632 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:44.504634 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:44.505027 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:38:44.505178 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:44.505184 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:44.505949 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:44.505952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:44.506617 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:44.506625 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:44.507038 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:44.507041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:44.507115 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:44.507656 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:38:44.507668 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:38:44.509097 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:38:44.510670 (+ 1573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:44.510678 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:44.510683 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:44.510894 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:44.511106 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:44.511108 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:44.511110 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:44.511183 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:44.511460 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:38:44.511465 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:38:44.511563 (+ 98us) client_negotiation.cc:769] Sending connection context
0927 02:38:44.511646 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 02:38:44.511749 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":82,"threads_started":1}
I20260927 02:38:44.511929 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:44.504048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33760 (local address 127.0.98.62:40591)
0927 02:38:44.504193 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:44.504198 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:44.504375 (+ 177us) server_negotiation.cc:407] Connection header received
0927 02:38:44.504481 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:44.504483 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:44.504524 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:44.504591 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:44.505327 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:44.505832 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:44.506720 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:44.506926 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:44.507002 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:44.509238 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:44.509241 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:44.509243 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:44.509263 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:44.510576 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:44.510977 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:44.510979 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:44.510980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:44.511045 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:44.511241 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:44.511243 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:44.511244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:44.511393 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:44.511459 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:44.511715 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 02:38:44.511808 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 02:38:44.515528 1508 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33760:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 02:38:44.516228 1508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:38:44.520814 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 05f7c5f1f9124c10b5b2a051f1cd2f7c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=530e990e5bb84dddb7449f0b54b07da4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:38:44.520812 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 05f7c5f1f9124c10b5b2a051f1cd2f7c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=530e990e5bb84dddb7449f0b54b07da4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:38:44.520841 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 05f7c5f1f9124c10b5b2a051f1cd2f7c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=530e990e5bb84dddb7449f0b54b07da4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:38:44.521196 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05f7c5f1f9124c10b5b2a051f1cd2f7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:38:44.521196 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05f7c5f1f9124c10b5b2a051f1cd2f7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:38:44.521243 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05f7c5f1f9124c10b5b2a051f1cd2f7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:38:44.523448 2014 tablet_bootstrap.cc:492] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 428bd355e94941368a1b1f8c964d5bac: Bootstrap starting.
I20260927 02:38:44.523929 2015 tablet_bootstrap.cc:492] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 8da60306942c4204a3c8bc44eb76b7fb: Bootstrap starting.
I20260927 02:38:44.524243 2014 tablet_bootstrap.cc:654] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 428bd355e94941368a1b1f8c964d5bac: Neither blocks nor log segments found. Creating new log.
I20260927 02:38:44.524439 2018 tablet_bootstrap.cc:492] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 90f8e9f42b7c4601944a51a87c1cf4cd: Bootstrap starting.
I20260927 02:38:44.524789 2015 tablet_bootstrap.cc:654] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 8da60306942c4204a3c8bc44eb76b7fb: Neither blocks nor log segments found. Creating new log.
I20260927 02:38:44.525354 2018 tablet_bootstrap.cc:654] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 90f8e9f42b7c4601944a51a87c1cf4cd: Neither blocks nor log segments found. Creating new log.
I20260927 02:38:44.525930 2015 tablet_bootstrap.cc:492] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 8da60306942c4204a3c8bc44eb76b7fb: No bootstrap required, opened a new log
I20260927 02:38:44.526008 2015 ts_tablet_manager.cc:1397] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 8da60306942c4204a3c8bc44eb76b7fb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:38:44.526132 2014 tablet_bootstrap.cc:492] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 428bd355e94941368a1b1f8c964d5bac: No bootstrap required, opened a new log
I20260927 02:38:44.526209 2014 ts_tablet_manager.cc:1397] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 428bd355e94941368a1b1f8c964d5bac: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 02:38:44.526448 2015 raft_consensus.cc:348] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 8da60306942c4204a3c8bc44eb76b7fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f8e9f42b7c4601944a51a87c1cf4cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40523 } } peers { permanent_uuid: "428bd355e94941368a1b1f8c964d5bac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39007 } } peers { permanent_uuid: "8da60306942c4204a3c8bc44eb76b7fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43353 } }
I20260927 02:38:44.526575 2015 raft_consensus.cc:374] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 8da60306942c4204a3c8bc44eb76b7fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:38:44.526615 2015 raft_consensus.cc:729] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 8da60306942c4204a3c8bc44eb76b7fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8da60306942c4204a3c8bc44eb76b7fb, State: Initialized, Role: FOLLOWER
I20260927 02:38:44.526641 2014 raft_consensus.cc:348] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 428bd355e94941368a1b1f8c964d5bac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f8e9f42b7c4601944a51a87c1cf4cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40523 } } peers { permanent_uuid: "428bd355e94941368a1b1f8c964d5bac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39007 } } peers { permanent_uuid: "8da60306942c4204a3c8bc44eb76b7fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43353 } }
I20260927 02:38:44.526746 2014 raft_consensus.cc:374] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 428bd355e94941368a1b1f8c964d5bac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:38:44.526742 2015 consensus_queue.cc:260] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 8da60306942c4204a3c8bc44eb76b7fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f8e9f42b7c4601944a51a87c1cf4cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40523 } } peers { permanent_uuid: "428bd355e94941368a1b1f8c964d5bac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39007 } } peers { permanent_uuid: "8da60306942c4204a3c8bc44eb76b7fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43353 } }
I20260927 02:38:44.526791 2014 raft_consensus.cc:729] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 428bd355e94941368a1b1f8c964d5bac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428bd355e94941368a1b1f8c964d5bac, State: Initialized, Role: FOLLOWER
I20260927 02:38:44.526896 2018 tablet_bootstrap.cc:492] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 90f8e9f42b7c4601944a51a87c1cf4cd: No bootstrap required, opened a new log
I20260927 02:38:44.526911 2014 consensus_queue.cc:260] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 428bd355e94941368a1b1f8c964d5bac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f8e9f42b7c4601944a51a87c1cf4cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40523 } } peers { permanent_uuid: "428bd355e94941368a1b1f8c964d5bac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39007 } } peers { permanent_uuid: "8da60306942c4204a3c8bc44eb76b7fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43353 } }
I20260927 02:38:44.526975 2018 ts_tablet_manager.cc:1397] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 90f8e9f42b7c4601944a51a87c1cf4cd: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260927 02:38:44.526991 2015 ts_tablet_manager.cc:1428] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 8da60306942c4204a3c8bc44eb76b7fb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:38:44.527169 2014 ts_tablet_manager.cc:1428] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 428bd355e94941368a1b1f8c964d5bac: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 02:38:44.527184 2020 tablet_replica.cc:439] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 8da60306942c4204a3c8bc44eb76b7fb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f8e9f42b7c4601944a51a87c1cf4cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40523 } } peers { permanent_uuid: "428bd355e94941368a1b1f8c964d5bac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39007 } } peers { permanent_uuid: "8da60306942c4204a3c8bc44eb76b7fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43353 } } }
I20260927 02:38:44.527307 2020 tablet_replica.cc:442] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 8da60306942c4204a3c8bc44eb76b7fb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:38:44.527333 2032 tablet_replica.cc:439] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 428bd355e94941368a1b1f8c964d5bac: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f8e9f42b7c4601944a51a87c1cf4cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40523 } } peers { permanent_uuid: "428bd355e94941368a1b1f8c964d5bac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39007 } } peers { permanent_uuid: "8da60306942c4204a3c8bc44eb76b7fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43353 } } }
I20260927 02:38:44.527381 2018 raft_consensus.cc:348] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 90f8e9f42b7c4601944a51a87c1cf4cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f8e9f42b7c4601944a51a87c1cf4cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40523 } } peers { permanent_uuid: "428bd355e94941368a1b1f8c964d5bac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39007 } } peers { permanent_uuid: "8da60306942c4204a3c8bc44eb76b7fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43353 } }
I20260927 02:38:44.527446 2032 tablet_replica.cc:442] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 428bd355e94941368a1b1f8c964d5bac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:38:44.527467 2018 raft_consensus.cc:374] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 90f8e9f42b7c4601944a51a87c1cf4cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:38:44.527495 2018 raft_consensus.cc:729] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 90f8e9f42b7c4601944a51a87c1cf4cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90f8e9f42b7c4601944a51a87c1cf4cd, State: Initialized, Role: FOLLOWER
I20260927 02:38:44.527596 2018 consensus_queue.cc:260] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 90f8e9f42b7c4601944a51a87c1cf4cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f8e9f42b7c4601944a51a87c1cf4cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40523 } } peers { permanent_uuid: "428bd355e94941368a1b1f8c964d5bac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39007 } } peers { permanent_uuid: "8da60306942c4204a3c8bc44eb76b7fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43353 } }
I20260927 02:38:44.527840 2018 ts_tablet_manager.cc:1428] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 90f8e9f42b7c4601944a51a87c1cf4cd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 02:38:44.528016 2022 tablet_replica.cc:439] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 90f8e9f42b7c4601944a51a87c1cf4cd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f8e9f42b7c4601944a51a87c1cf4cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40523 } } peers { permanent_uuid: "428bd355e94941368a1b1f8c964d5bac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39007 } } peers { permanent_uuid: "8da60306942c4204a3c8bc44eb76b7fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43353 } } }
I20260927 02:38:44.528147 2022 tablet_replica.cc:442] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 90f8e9f42b7c4601944a51a87c1cf4cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:38:44.557247 2032 raft_consensus.cc:482] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 428bd355e94941368a1b1f8c964d5bac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:38:44.557372 2032 raft_consensus.cc:504] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 428bd355e94941368a1b1f8c964d5bac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f8e9f42b7c4601944a51a87c1cf4cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40523 } } peers { permanent_uuid: "428bd355e94941368a1b1f8c964d5bac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39007 } } peers { permanent_uuid: "8da60306942c4204a3c8bc44eb76b7fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43353 } }
I20260927 02:38:44.557758 2032 leader_election.cc:288] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 428bd355e94941368a1b1f8c964d5bac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90f8e9f42b7c4601944a51a87c1cf4cd (127.0.98.1:40523), 8da60306942c4204a3c8bc44eb76b7fb (127.0.98.3:43353)
I20260927 02:38:44.558209 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f7c5f1f9124c10b5b2a051f1cd2f7c" candidate_uuid: "428bd355e94941368a1b1f8c964d5bac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8da60306942c4204a3c8bc44eb76b7fb" is_pre_election: true
I20260927 02:38:44.558207 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f7c5f1f9124c10b5b2a051f1cd2f7c" candidate_uuid: "428bd355e94941368a1b1f8c964d5bac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90f8e9f42b7c4601944a51a87c1cf4cd" is_pre_election: true
I20260927 02:38:44.558382 1945 raft_consensus.cc:2454] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 8da60306942c4204a3c8bc44eb76b7fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428bd355e94941368a1b1f8c964d5bac in term 0.
I20260927 02:38:44.558382 1665 raft_consensus.cc:2454] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 90f8e9f42b7c4601944a51a87c1cf4cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428bd355e94941368a1b1f8c964d5bac in term 0.
I20260927 02:38:44.558660 1737 leader_election.cc:302] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 428bd355e94941368a1b1f8c964d5bac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 428bd355e94941368a1b1f8c964d5bac, 8da60306942c4204a3c8bc44eb76b7fb, 90f8e9f42b7c4601944a51a87c1cf4cd; no voters:
I20260927 02:38:44.558843 2032 raft_consensus.cc:2781] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 428bd355e94941368a1b1f8c964d5bac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:38:44.558905 2032 raft_consensus.cc:482] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 428bd355e94941368a1b1f8c964d5bac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:38:44.558930 2032 raft_consensus.cc:3037] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 428bd355e94941368a1b1f8c964d5bac [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:38:44.559875 2032 raft_consensus.cc:504] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 428bd355e94941368a1b1f8c964d5bac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f8e9f42b7c4601944a51a87c1cf4cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40523 } } peers { permanent_uuid: "428bd355e94941368a1b1f8c964d5bac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39007 } } peers { permanent_uuid: "8da60306942c4204a3c8bc44eb76b7fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43353 } }
I20260927 02:38:44.560199 2032 leader_election.cc:288] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 428bd355e94941368a1b1f8c964d5bac [CANDIDATE]: Term 1 election: Requested vote from peers 90f8e9f42b7c4601944a51a87c1cf4cd (127.0.98.1:40523), 8da60306942c4204a3c8bc44eb76b7fb (127.0.98.3:43353)
I20260927 02:38:44.560554 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f7c5f1f9124c10b5b2a051f1cd2f7c" candidate_uuid: "428bd355e94941368a1b1f8c964d5bac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8da60306942c4204a3c8bc44eb76b7fb"
I20260927 02:38:44.560684 1945 raft_consensus.cc:3037] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 8da60306942c4204a3c8bc44eb76b7fb [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:38:44.560686 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f7c5f1f9124c10b5b2a051f1cd2f7c" candidate_uuid: "428bd355e94941368a1b1f8c964d5bac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90f8e9f42b7c4601944a51a87c1cf4cd"
I20260927 02:38:44.560783 1665 raft_consensus.cc:3037] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 90f8e9f42b7c4601944a51a87c1cf4cd [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:38:44.561779 1945 raft_consensus.cc:2454] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 8da60306942c4204a3c8bc44eb76b7fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428bd355e94941368a1b1f8c964d5bac in term 1.
I20260927 02:38:44.561779 1665 raft_consensus.cc:2454] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 90f8e9f42b7c4601944a51a87c1cf4cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428bd355e94941368a1b1f8c964d5bac in term 1.
I20260927 02:38:44.562047 1737 leader_election.cc:302] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 428bd355e94941368a1b1f8c964d5bac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428bd355e94941368a1b1f8c964d5bac, 90f8e9f42b7c4601944a51a87c1cf4cd; no voters:
I20260927 02:38:44.562320 2032 raft_consensus.cc:2781] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 428bd355e94941368a1b1f8c964d5bac [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:38:44.562409 2032 raft_consensus.cc:686] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 428bd355e94941368a1b1f8c964d5bac [term 1 LEADER]: Becoming Leader. State: Replica: 428bd355e94941368a1b1f8c964d5bac, State: Running, Role: LEADER
I20260927 02:38:44.562546 2032 consensus_queue.cc:237] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 428bd355e94941368a1b1f8c964d5bac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f8e9f42b7c4601944a51a87c1cf4cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40523 } } peers { permanent_uuid: "428bd355e94941368a1b1f8c964d5bac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39007 } } peers { permanent_uuid: "8da60306942c4204a3c8bc44eb76b7fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43353 } }
I20260927 02:38:44.563680 2024 tablet_replica.cc:439] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 428bd355e94941368a1b1f8c964d5bac: TxnStatusTablet state changed. Reason: New leader 428bd355e94941368a1b1f8c964d5bac. Latest consensus state: current_term: 1 leader_uuid: "428bd355e94941368a1b1f8c964d5bac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f8e9f42b7c4601944a51a87c1cf4cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40523 } } peers { permanent_uuid: "428bd355e94941368a1b1f8c964d5bac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39007 } } peers { permanent_uuid: "8da60306942c4204a3c8bc44eb76b7fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43353 } } }
I20260927 02:38:44.563793 2024 tablet_replica.cc:442] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 428bd355e94941368a1b1f8c964d5bac: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:38:44.563998 2054 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 02:38:44.564155 2054 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 02:38:44.564126 1506 catalog_manager.cc:5495] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 428bd355e94941368a1b1f8c964d5bac reported cstate change: term changed from 0 to 1, leader changed from <none> to 428bd355e94941368a1b1f8c964d5bac (127.0.98.2). New cstate: current_term: 1 leader_uuid: "428bd355e94941368a1b1f8c964d5bac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f8e9f42b7c4601944a51a87c1cf4cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "428bd355e94941368a1b1f8c964d5bac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8da60306942c4204a3c8bc44eb76b7fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43353 } health_report { overall_health: UNKNOWN } } }
I20260927 02:38:44.580376 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:44.576982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41120 (local address 127.0.98.2:39007)
0927 02:38:44.577103 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:44.577107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:44.577123 (+ 16us) server_negotiation.cc:407] Connection header received
0927 02:38:44.577329 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:44.577332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:44.577377 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:44.577444 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:38:44.577673 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:44.578175 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:44.579349 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:44.579574 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:44.579656 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:44.579941 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:38:44.580009 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:38:44.580056 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:44.580159 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 02:38:44.580236 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 02:38:44.580377 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:44.576878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39007 (local address 127.0.0.1:41120)
0927 02:38:44.577054 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:44.577069 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:44.577177 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:44.577469 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:44.577471 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:44.577481 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:38:44.577582 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:44.577587 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:44.578284 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:44.578287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:44.579242 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:44.579251 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:44.579668 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:44.579672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:44.579760 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:44.579814 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:38:44.580071 (+ 257us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:38:44.580089 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 02:38:44.580152 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 02:38:44.580238 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 02:38:44.582800 1665 raft_consensus.cc:1261] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 90f8e9f42b7c4601944a51a87c1cf4cd [term 1 FOLLOWER]: Refusing update from remote peer 428bd355e94941368a1b1f8c964d5bac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:38:44.582818 1945 raft_consensus.cc:1261] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 8da60306942c4204a3c8bc44eb76b7fb [term 1 FOLLOWER]: Refusing update from remote peer 428bd355e94941368a1b1f8c964d5bac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:38:44.583230 2024 consensus_queue.cc:1035] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 428bd355e94941368a1b1f8c964d5bac [LEADER]: Connected to new peer: Peer: permanent_uuid: "90f8e9f42b7c4601944a51a87c1cf4cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:38:44.583382 2032 consensus_queue.cc:1035] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 428bd355e94941368a1b1f8c964d5bac [LEADER]: Connected to new peer: Peer: permanent_uuid: "8da60306942c4204a3c8bc44eb76b7fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:38:44.584632 2020 tablet_replica.cc:439] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 8da60306942c4204a3c8bc44eb76b7fb: TxnStatusTablet state changed. Reason: New leader 428bd355e94941368a1b1f8c964d5bac. Latest consensus state: current_term: 1 leader_uuid: "428bd355e94941368a1b1f8c964d5bac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f8e9f42b7c4601944a51a87c1cf4cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40523 } } peers { permanent_uuid: "428bd355e94941368a1b1f8c964d5bac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39007 } } peers { permanent_uuid: "8da60306942c4204a3c8bc44eb76b7fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43353 } } }
I20260927 02:38:44.584753 2020 tablet_replica.cc:442] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 8da60306942c4204a3c8bc44eb76b7fb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:38:44.585913 2022 tablet_replica.cc:439] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 90f8e9f42b7c4601944a51a87c1cf4cd: TxnStatusTablet state changed. Reason: New leader 428bd355e94941368a1b1f8c964d5bac. Latest consensus state: current_term: 1 leader_uuid: "428bd355e94941368a1b1f8c964d5bac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f8e9f42b7c4601944a51a87c1cf4cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40523 } } peers { permanent_uuid: "428bd355e94941368a1b1f8c964d5bac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39007 } } peers { permanent_uuid: "8da60306942c4204a3c8bc44eb76b7fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43353 } } }
I20260927 02:38:44.586025 2022 tablet_replica.cc:442] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 90f8e9f42b7c4601944a51a87c1cf4cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:38:44.586927 2032 tablet_replica.cc:439] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 428bd355e94941368a1b1f8c964d5bac: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "428bd355e94941368a1b1f8c964d5bac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f8e9f42b7c4601944a51a87c1cf4cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40523 } } peers { permanent_uuid: "428bd355e94941368a1b1f8c964d5bac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39007 } } peers { permanent_uuid: "8da60306942c4204a3c8bc44eb76b7fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43353 } } }
I20260927 02:38:44.587039 2032 tablet_replica.cc:442] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 428bd355e94941368a1b1f8c964d5bac: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:38:44.587124 2027 mvcc.cc:205] Tried to move back new op lower bound from 7333792663887314944 to 7333792663809462272. Current Snapshot: MvccSnapshot[applied={T|T < 7333792663887314944}]
I20260927 02:38:44.587965 2019 tablet_replica.cc:439] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 428bd355e94941368a1b1f8c964d5bac: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "428bd355e94941368a1b1f8c964d5bac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f8e9f42b7c4601944a51a87c1cf4cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40523 } } peers { permanent_uuid: "428bd355e94941368a1b1f8c964d5bac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39007 } } peers { permanent_uuid: "8da60306942c4204a3c8bc44eb76b7fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43353 } } }
I20260927 02:38:44.588058 2019 tablet_replica.cc:442] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 428bd355e94941368a1b1f8c964d5bac: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:38:44.588779 2028 mvcc.cc:205] Tried to move back new op lower bound from 7333792663887314944 to 7333792663809462272. Current Snapshot: MvccSnapshot[applied={T|T < 7333792663887314944}]
I20260927 02:38:44.589231 2022 tablet_replica.cc:439] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 90f8e9f42b7c4601944a51a87c1cf4cd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "428bd355e94941368a1b1f8c964d5bac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f8e9f42b7c4601944a51a87c1cf4cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40523 } } peers { permanent_uuid: "428bd355e94941368a1b1f8c964d5bac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39007 } } peers { permanent_uuid: "8da60306942c4204a3c8bc44eb76b7fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43353 } } }
I20260927 02:38:44.589334 2022 tablet_replica.cc:442] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 90f8e9f42b7c4601944a51a87c1cf4cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:38:44.589845 2020 tablet_replica.cc:439] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 8da60306942c4204a3c8bc44eb76b7fb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "428bd355e94941368a1b1f8c964d5bac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90f8e9f42b7c4601944a51a87c1cf4cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40523 } } peers { permanent_uuid: "428bd355e94941368a1b1f8c964d5bac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39007 } } peers { permanent_uuid: "8da60306942c4204a3c8bc44eb76b7fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43353 } } }
I20260927 02:38:44.589959 2020 tablet_replica.cc:442] T 05f7c5f1f9124c10b5b2a051f1cd2f7c P 8da60306942c4204a3c8bc44eb76b7fb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:38:44.604036 2060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:44.600308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39007 (local address 127.0.98.2:56043)
0927 02:38:44.600627 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:44.600638 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:44.600738 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:44.600938 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:44.600940 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:44.600945 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:38:44.601053 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:44.601058 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:44.602015 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:44.602018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:44.602756 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:44.602765 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:44.603355 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:44.603358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:44.603458 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:44.603499 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:38:44.603796 (+ 297us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:38:44.603800 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:38:44.603844 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 02:38:44.603906 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":47,"thread_start_us":110,"threads_started":1}
I20260927 02:38:44.604048 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:44.600397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56043 (local address 127.0.98.2:39007)
0927 02:38:44.600513 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:44.600518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:44.600714 (+ 196us) server_negotiation.cc:407] Connection header received
0927 02:38:44.600805 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:44.600808 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:44.600854 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:44.600934 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:38:44.601133 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:44.601909 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:44.602899 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:44.603230 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:44.603342 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:44.603616 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:38:44.603694 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:38:44.603796 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:44.603863 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 02:38:44.603912 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":43}
I20260927 02:38:44.610328 1665 raft_consensus.cc:1261] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 90f8e9f42b7c4601944a51a87c1cf4cd [term 1 FOLLOWER]: Refusing update from remote peer 428bd355e94941368a1b1f8c964d5bac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:38:44.610433 1945 raft_consensus.cc:1261] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 8da60306942c4204a3c8bc44eb76b7fb [term 1 FOLLOWER]: Refusing update from remote peer 428bd355e94941368a1b1f8c964d5bac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:38:44.610787 2024 consensus_queue.cc:1035] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 428bd355e94941368a1b1f8c964d5bac [LEADER]: Connected to new peer: Peer: permanent_uuid: "8da60306942c4204a3c8bc44eb76b7fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:38:44.610929 2032 consensus_queue.cc:1035] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 428bd355e94941368a1b1f8c964d5bac [LEADER]: Connected to new peer: Peer: permanent_uuid: "90f8e9f42b7c4601944a51a87c1cf4cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:38:44.645044 1508 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:38:44.645184 1508 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:38:44.646997 1508 catalog_manager.cc:5782] T 00000000000000000000000000000000 P edff1a4c13434b2e9d0ddfc6409df266: Sending DeleteTablet for 3 replicas of tablet 17d4e7de7b6f43cbafd2706d19a04cb5
I20260927 02:38:44.647442 1645 tablet_service.cc:1514] Processing DeleteTablet for tablet 17d4e7de7b6f43cbafd2706d19a04cb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:38:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53376
I20260927 02:38:44.647524 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet 17d4e7de7b6f43cbafd2706d19a04cb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:38:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54790
I20260927 02:38:44.647614 2037 tablet_replica.cc:330] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 90f8e9f42b7c4601944a51a87c1cf4cd: stopping tablet replica
I20260927 02:38:44.647688 2039 tablet_replica.cc:330] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 8da60306942c4204a3c8bc44eb76b7fb: stopping tablet replica
I20260927 02:38:44.647715 2037 raft_consensus.cc:2229] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 90f8e9f42b7c4601944a51a87c1cf4cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:38:44.647778 2039 raft_consensus.cc:2229] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 8da60306942c4204a3c8bc44eb76b7fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:38:44.647828 2037 raft_consensus.cc:2258] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 90f8e9f42b7c4601944a51a87c1cf4cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:38:44.647876 2039 raft_consensus.cc:2258] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 8da60306942c4204a3c8bc44eb76b7fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:38:44.648087 1785 tablet_service.cc:1514] Processing DeleteTablet for tablet 17d4e7de7b6f43cbafd2706d19a04cb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:38:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41108
I20260927 02:38:44.648231 2038 tablet_replica.cc:330] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 428bd355e94941368a1b1f8c964d5bac: stopping tablet replica
I20260927 02:38:44.648317 2038 raft_consensus.cc:2229] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 428bd355e94941368a1b1f8c964d5bac [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:38:44.648464 2038 raft_consensus.cc:2258] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 428bd355e94941368a1b1f8c964d5bac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:38:44.648505 2037 ts_tablet_manager.cc:1905] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 90f8e9f42b7c4601944a51a87c1cf4cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:38:44.650022 2038 ts_tablet_manager.cc:1905] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 428bd355e94941368a1b1f8c964d5bac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:38:44.651094 2037 ts_tablet_manager.cc:1918] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 90f8e9f42b7c4601944a51a87c1cf4cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:38:44.651160 2037 log.cc:1196] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 90f8e9f42b7c4601944a51a87c1cf4cd: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-0/wal/wals/17d4e7de7b6f43cbafd2706d19a04cb5
I20260927 02:38:44.651360 2037 ts_tablet_manager.cc:1939] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 90f8e9f42b7c4601944a51a87c1cf4cd: Deleting consensus metadata
I20260927 02:38:44.651957 1493 catalog_manager.cc:4841] TS 90f8e9f42b7c4601944a51a87c1cf4cd (127.0.98.1:40523): tablet 17d4e7de7b6f43cbafd2706d19a04cb5 (table test-table [id=745c5d073dd445189dd4dd2976cf705d]) successfully deleted
I20260927 02:38:44.652596 2038 ts_tablet_manager.cc:1918] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 428bd355e94941368a1b1f8c964d5bac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:38:44.652671 2038 log.cc:1196] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 428bd355e94941368a1b1f8c964d5bac: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-1/wal/wals/17d4e7de7b6f43cbafd2706d19a04cb5
I20260927 02:38:44.652905 2038 ts_tablet_manager.cc:1939] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 428bd355e94941368a1b1f8c964d5bac: Deleting consensus metadata
I20260927 02:38:44.653463 1495 catalog_manager.cc:4841] TS 428bd355e94941368a1b1f8c964d5bac (127.0.98.2:39007): tablet 17d4e7de7b6f43cbafd2706d19a04cb5 (table test-table [id=745c5d073dd445189dd4dd2976cf705d]) successfully deleted
I20260927 02:38:44.654295 2039 ts_tablet_manager.cc:1905] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 8da60306942c4204a3c8bc44eb76b7fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:38:44.656765 2039 ts_tablet_manager.cc:1918] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 8da60306942c4204a3c8bc44eb76b7fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:38:44.656836 2039 log.cc:1196] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 8da60306942c4204a3c8bc44eb76b7fb: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790476718023131-392-0/minicluster-data/ts-2/wal/wals/17d4e7de7b6f43cbafd2706d19a04cb5
I20260927 02:38:44.657043 2039 ts_tablet_manager.cc:1939] T 17d4e7de7b6f43cbafd2706d19a04cb5 P 8da60306942c4204a3c8bc44eb76b7fb: Deleting consensus metadata
I20260927 02:38:44.657624 1494 catalog_manager.cc:4841] TS 8da60306942c4204a3c8bc44eb76b7fb (127.0.98.3:43353): tablet 17d4e7de7b6f43cbafd2706d19a04cb5 (table test-table [id=745c5d073dd445189dd4dd2976cf705d]) successfully deleted
Sep 27 02:38:44 dist-test-slave-zv8z krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476724, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:38:44.662670 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:44.649758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53392 (local address 127.0.98.1:40523)
0927 02:38:44.653449 (+ 3691us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:44.653453 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:44.653468 (+ 15us) server_negotiation.cc:407] Connection header received
0927 02:38:44.653503 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:44.653506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:44.653548 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:44.653627 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:44.654360 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:44.654864 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:44.655663 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:44.655883 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:44.655967 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:44.659213 (+ 3246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:44.659226 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:44.659243 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:44.659269 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:44.660893 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:44.661385 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:44.661391 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:44.661392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:44.661489 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:44.661751 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:44.661754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:44.661756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:44.662015 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:44.662213 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:44.662394 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 02:38:44.662490 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3616}
W20260927 02:38:44.663036 1705 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53392
I20260927 02:38:44.671240 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:44.664609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33766 (local address 127.0.98.62:40591)
0927 02:38:44.664727 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:44.664731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:44.664812 (+ 81us) server_negotiation.cc:407] Connection header received
0927 02:38:44.664915 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:44.664918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:44.664962 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:44.665028 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:44.665629 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:44.666145 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:44.666909 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:44.667130 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:44.667212 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:44.668438 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:44.668441 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:44.668444 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:44.668469 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:44.669723 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:44.670085 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:44.670089 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:44.670091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:44.670167 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:44.670486 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:44.670490 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:44.670491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:44.670657 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:44.670783 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:44.670996 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 02:38:44.671098 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260927 02:38:44.671563 1508 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33766
I20260927 02:38:44.672788 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 1580
I20260927 02:38:44.678742 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 1720
I20260927 02:38:44.684588 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 1860
I20260927 02:38:44.688972 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 1476
2026-09-27T02:38:44Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2441 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:38:44 dist-test-slave-zv8z krb5kdc[2070](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:38:44 dist-test-slave-zv8z krb5kdc[2070](info): set up 2 sockets
Sep 27 02:38:44 dist-test-slave-zv8z krb5kdc[2070](info): commencing operation
krb5kdc: starting...
W20260927 02:38:45.629866 392 mini_kdc.cc:121] Time spent starting KDC: real 0.921s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/krb5kdc/test-user.keytab.
Sep 27 02:38:45 dist-test-slave-zv8z krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:38:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:38:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:38:45.744798 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33095
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33095
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:38:45.849471 2083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:38:45.849735 2083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:38:45.849787 2083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:38:45.853754 2083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:38:45.853809 2083 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 02:38:45.853832 2083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:38:45.853854 2083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:38:45.853875 2083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:38:45.853900 2083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:38:45.859069 2083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33095
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:38:45.860070 2083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:38:45.861039 2083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:38:45.868917 2088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:38:45.868949 2091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:38:45.868963 2089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:38:45.869019 2083 server_base.cc:1034] running on GCE node
I20260927 02:38:45.869778 2083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:38:45.870863 2083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:38:45.872035 2083 hybrid_clock.cc:648] HybridClock initialized: now 1790476725872020 us; error 32 us; skew 500 ppm
Sep 27 02:38:45 dist-test-slave-zv8z krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:38:45.874766 2083 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:38:45.875609 2083 webserver.cc:460] Webserver started at http://127.0.98.62:46267/ using document root <none> and password file <none>
I20260927 02:38:45.876120 2083 fs_manager.cc:359] Metadata directory not provided
I20260927 02:38:45.876181 2083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:38:45.876394 2083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:38:45.878440 2083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/master-0/data/instance:
uuid: "336ef4719fc24eb4aeb74a965882da30"
format_stamp: "Formatted at 2026-09-27 02:38:45 on dist-test-slave-zv8z"
server_key: "07e2a29c9de8868aa3b1805f69569810"
server_key_iv: "fd178e3d26ce41e92cebaf81c019709b"
server_key_version: "encryptionkey@0"
I20260927 02:38:45.878970 2083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/master-0/wal/instance:
uuid: "336ef4719fc24eb4aeb74a965882da30"
format_stamp: "Formatted at 2026-09-27 02:38:45 on dist-test-slave-zv8z"
server_key: "07e2a29c9de8868aa3b1805f69569810"
server_key_iv: "fd178e3d26ce41e92cebaf81c019709b"
server_key_version: "encryptionkey@0"
I20260927 02:38:45.882539 2083 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:38:45.884981 2098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:45.886332 2083 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:38:45.886444 2083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "336ef4719fc24eb4aeb74a965882da30"
format_stamp: "Formatted at 2026-09-27 02:38:45 on dist-test-slave-zv8z"
server_key: "07e2a29c9de8868aa3b1805f69569810"
server_key_iv: "fd178e3d26ce41e92cebaf81c019709b"
server_key_version: "encryptionkey@0"
I20260927 02:38:45.886538 2083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:38:45.899318 2083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:38:45.902420 2083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:38:45.902590 2083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:38:45.909673 2083 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33095
I20260927 02:38:45.909716 2160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33095 every 8 connection(s)
I20260927 02:38:45.910612 2083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:38:45.913259 2161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:38:45.918411 2161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 336ef4719fc24eb4aeb74a965882da30: Bootstrap starting.
I20260927 02:38:45.920325 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 2083
I20260927 02:38:45.920420 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/master-0/wal/instance
I20260927 02:38:45.920643 392 external_mini_cluster.cc:1428] Setting key 2dc888b6b7c2aca0899baa75437cb23a
I20260927 02:38:45.920799 2161 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 336ef4719fc24eb4aeb74a965882da30: Neither blocks nor log segments found. Creating new log.
I20260927 02:38:45.921517 2161 log.cc:824] T 00000000000000000000000000000000 P 336ef4719fc24eb4aeb74a965882da30: Log is configured to *not* fsync() on all Append() calls
I20260927 02:38:45.923391 2161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 336ef4719fc24eb4aeb74a965882da30: No bootstrap required, opened a new log
I20260927 02:38:45.927347 2161 raft_consensus.cc:348] T 00000000000000000000000000000000 P 336ef4719fc24eb4aeb74a965882da30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "336ef4719fc24eb4aeb74a965882da30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33095 } }
I20260927 02:38:45.927596 2161 raft_consensus.cc:374] T 00000000000000000000000000000000 P 336ef4719fc24eb4aeb74a965882da30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:38:45.927634 2161 raft_consensus.cc:729] T 00000000000000000000000000000000 P 336ef4719fc24eb4aeb74a965882da30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 336ef4719fc24eb4aeb74a965882da30, State: Initialized, Role: FOLLOWER
I20260927 02:38:45.928105 2161 consensus_queue.cc:260] T 00000000000000000000000000000000 P 336ef4719fc24eb4aeb74a965882da30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "336ef4719fc24eb4aeb74a965882da30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33095 } }
I20260927 02:38:45.928277 2161 raft_consensus.cc:388] T 00000000000000000000000000000000 P 336ef4719fc24eb4aeb74a965882da30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:38:45.928347 2161 raft_consensus.cc:482] T 00000000000000000000000000000000 P 336ef4719fc24eb4aeb74a965882da30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:38:45.928448 2161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 336ef4719fc24eb4aeb74a965882da30 [term 0 FOLLOWER]: Advancing to term 1
Sep 27 02:38:45 dist-test-slave-zv8z krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:38:45.929380 2161 raft_consensus.cc:504] T 00000000000000000000000000000000 P 336ef4719fc24eb4aeb74a965882da30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "336ef4719fc24eb4aeb74a965882da30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33095 } }
I20260927 02:38:45.929742 2161 leader_election.cc:302] T 00000000000000000000000000000000 P 336ef4719fc24eb4aeb74a965882da30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 336ef4719fc24eb4aeb74a965882da30; no voters:
I20260927 02:38:45.930081 2161 leader_election.cc:288] T 00000000000000000000000000000000 P 336ef4719fc24eb4aeb74a965882da30 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:38:45.930181 2166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 336ef4719fc24eb4aeb74a965882da30 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:38:45.930711 2166 raft_consensus.cc:686] T 00000000000000000000000000000000 P 336ef4719fc24eb4aeb74a965882da30 [term 1 LEADER]: Becoming Leader. State: Replica: 336ef4719fc24eb4aeb74a965882da30, State: Running, Role: LEADER
I20260927 02:38:45.931257 2161 sys_catalog.cc:564] T 00000000000000000000000000000000 P 336ef4719fc24eb4aeb74a965882da30 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:38:45.931299 2166 consensus_queue.cc:237] T 00000000000000000000000000000000 P 336ef4719fc24eb4aeb74a965882da30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "336ef4719fc24eb4aeb74a965882da30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33095 } }
I20260927 02:38:45.933229 2167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 336ef4719fc24eb4aeb74a965882da30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "336ef4719fc24eb4aeb74a965882da30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "336ef4719fc24eb4aeb74a965882da30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33095 } } }
I20260927 02:38:45.933491 2167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 336ef4719fc24eb4aeb74a965882da30 [sys.catalog]: This master's current role is: LEADER
I20260927 02:38:45.933115 2168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 336ef4719fc24eb4aeb74a965882da30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 336ef4719fc24eb4aeb74a965882da30. Latest consensus state: current_term: 1 leader_uuid: "336ef4719fc24eb4aeb74a965882da30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "336ef4719fc24eb4aeb74a965882da30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33095 } } }
I20260927 02:38:45.933885 2168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 336ef4719fc24eb4aeb74a965882da30 [sys.catalog]: This master's current role is: LEADER
I20260927 02:38:45.935478 2165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:45.921669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57170 (local address 127.0.98.62:33095)
0927 02:38:45.922053 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:45.922069 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:45.922104 (+ 35us) server_negotiation.cc:407] Connection header received
0927 02:38:45.922782 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:45.922798 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:45.923110 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:45.923392 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:45.924321 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:45.925327 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:45.926535 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:45.927019 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:45.927136 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:45.929979 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:45.929984 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:45.929996 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:45.930022 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:45.932135 (+ 2113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:45.932620 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:45.932626 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:45.932632 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:45.932766 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:45.933094 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:45.933098 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:45.933100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:45.933649 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:45.933843 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:45.934283 (+ 440us) server_negotiation.cc:299] Negotiation successful
0927 02:38:45.934571 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":105,"threads_started":1}
W20260927 02:38:45.936228 2180 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 336ef4719fc24eb4aeb74a965882da30: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:38:45.936293 2180 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:38:45.936365 2181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:38:45.937168 2181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:38:45.942231 2181 catalog_manager.cc:1347] Generated new cluster ID: 77a7b537283b434198f3af21e1d8b836
I20260927 02:38:45.942317 2181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:38:45.955585 2181 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:38:45.956848 2181 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:38:45.976936 2181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 336ef4719fc24eb4aeb74a965882da30: Generated new TSK 0
I20260927 02:38:45.977645 2181 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:38:46.011315 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33095
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:38:46.118844 2187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:38:46.119103 2187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:38:46.119149 2187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:38:46.123075 2187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:38:46.123131 2187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:38:46.123168 2187 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 02:38:46.123180 2187 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:38:46.123211 2187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:38:46.128338 2187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:38:46.129379 2187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:38:46.130446 2187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:38:46.139864 2193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:38:46.139840 2192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:38:46.139822 2195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:38:46.140060 2187 server_base.cc:1034] running on GCE node
I20260927 02:38:46.140492 2187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:38:46.141099 2187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:38:46.142343 2187 hybrid_clock.cc:648] HybridClock initialized: now 1790476726142316 us; error 37 us; skew 500 ppm
Sep 27 02:38:46 dist-test-slave-zv8z krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:38:46.145105 2187 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:38:46.146010 2187 webserver.cc:460] Webserver started at http://127.0.98.1:40531/ using document root <none> and password file <none>
I20260927 02:38:46.146598 2187 fs_manager.cc:359] Metadata directory not provided
I20260927 02:38:46.146659 2187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:38:46.146823 2187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:38:46.148783 2187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-0/data/instance:
uuid: "bfd99789611f4ab098a248d6366e52c9"
format_stamp: "Formatted at 2026-09-27 02:38:46 on dist-test-slave-zv8z"
server_key: "88ca7ad9030880115d72f5081a08f171"
server_key_iv: "5772897d981e48786d321b85fd29ea91"
server_key_version: "encryptionkey@0"
I20260927 02:38:46.149277 2187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bfd99789611f4ab098a248d6366e52c9"
format_stamp: "Formatted at 2026-09-27 02:38:46 on dist-test-slave-zv8z"
server_key: "88ca7ad9030880115d72f5081a08f171"
server_key_iv: "5772897d981e48786d321b85fd29ea91"
server_key_version: "encryptionkey@0"
I20260927 02:38:46.152940 2187 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 02:38:46.155395 2202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:46.156782 2187 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:38:46.156884 2187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-0/wal
uuid: "bfd99789611f4ab098a248d6366e52c9"
format_stamp: "Formatted at 2026-09-27 02:38:46 on dist-test-slave-zv8z"
server_key: "88ca7ad9030880115d72f5081a08f171"
server_key_iv: "5772897d981e48786d321b85fd29ea91"
server_key_version: "encryptionkey@0"
I20260927 02:38:46.156991 2187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:38:46.168955 2187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:38:46.171895 2187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:38:46.172057 2187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:38:46.173431 2187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:38:46.173481 2187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:46.173533 2187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:38:46.173559 2187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 02:38:46 dist-test-slave-zv8z krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:38:46.183585 2317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43257 every 8 connection(s)
I20260927 02:38:46.183367 2187 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43257
I20260927 02:38:46.184717 2187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:38:46.186904 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 2187
I20260927 02:38:46.187021 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:38:46.187330 392 external_mini_cluster.cc:1428] Setting key a2e050f32922aa3b7758df223022db5b
I20260927 02:38:46.187588 2165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:46.174189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34909 (local address 127.0.98.62:33095)
0927 02:38:46.174358 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:46.174364 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:46.175227 (+ 863us) server_negotiation.cc:407] Connection header received
0927 02:38:46.176445 (+ 1218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:46.176449 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:46.176509 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:46.176568 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:46.178332 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:46.178999 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:46.180638 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:46.180863 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:46.180945 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:46.183952 (+ 3007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:46.183955 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:46.183957 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:46.183987 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:46.185568 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:46.186036 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:46.186040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:46.186041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:46.186123 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:46.186468 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:46.186472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:46.186473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:46.186674 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:46.186756 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:46.187288 (+ 532us) server_negotiation.cc:299] Negotiation successful
0927 02:38:46.187429 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 02:38:46.188431 2211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:46.174569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33095 (local address 127.0.98.1:34909)
0927 02:38:46.175046 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:46.175107 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:46.176162 (+ 1055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:46.176993 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:46.177004 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:46.177450 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:38:46.178137 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:46.178160 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:46.179108 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:46.179112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:46.180506 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:46.180519 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:46.181025 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:46.181029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:46.181234 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:46.181900 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:38:46.181916 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:38:46.183793 (+ 1877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:38:46.185671 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:46.185677 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:46.185692 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:46.185933 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:46.186202 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:46.186205 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:46.186207 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:46.186374 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:46.186775 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:38:46.186781 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:38:46.187016 (+ 235us) client_negotiation.cc:769] Sending connection context
0927 02:38:46.187201 (+ 185us) client_negotiation.cc:240] Negotiation successful
0927 02:38:46.187507 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":102,"threads_started":1}
I20260927 02:38:46.189746 2318 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33095
I20260927 02:38:46.189901 2318 heartbeater.cc:464] Registering TS with master...
I20260927 02:38:46.190562 2318 heartbeater.cc:511] Master 127.0.98.62:33095 requested a full tablet report, sending...
I20260927 02:38:46.192198 2114 ts_manager.cc:194] Registered new tserver with Master: bfd99789611f4ab098a248d6366e52c9 (127.0.98.1:43257)
I20260927 02:38:46.193364 2114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34909
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:38:46.200162 2165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:46.191902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42493 (local address 127.0.98.62:33095)
0927 02:38:46.193324 (+ 1422us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:46.193327 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:46.193339 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:38:46.193370 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:46.193372 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:46.193406 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:46.193483 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:46.193992 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:46.194693 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:46.195931 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:46.196149 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:46.196248 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:46.197412 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:46.197414 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:46.197416 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:46.197438 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:46.198733 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:46.199109 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:46.199111 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:46.199113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:46.199176 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:46.199496 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:46.199498 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:46.199499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:46.199673 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:46.199746 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:46.199961 (+ 215us) server_negotiation.cc:299] Negotiation successful
0927 02:38:46.200052 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1337}
I20260927 02:38:46.200171 2325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:46.191839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33095 (local address 127.0.98.1:42493)
0927 02:38:46.192104 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:46.192118 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:46.192199 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:46.193524 (+ 1325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:46.193527 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:46.193735 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:38:46.193864 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:46.193872 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:46.194829 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:46.194833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:46.195790 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:46.195802 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:46.196272 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:46.196277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:46.196369 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:46.196840 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:38:46.196854 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:38:46.197276 (+ 422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:38:46.198847 (+ 1571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:46.198850 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:46.198851 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:46.199035 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:46.199270 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:46.199273 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:46.199275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:46.199366 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:46.199758 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:38:46.199761 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:38:46.199838 (+ 77us) client_negotiation.cc:769] Sending connection context
0927 02:38:46.199920 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 02:38:46.200033 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":24,"thread_start_us":106,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:38:46.214905 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33095
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:38:46.320991 2327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:38:46.321280 2327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:38:46.321349 2327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:38:46.325348 2327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:38:46.325409 2327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:38:46.325464 2327 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 02:38:46.325490 2327 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:38:46.325538 2327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:38:46.330835 2327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:38:46.331869 2327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:38:46.332924 2327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:38:46.341739 2335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:38:46.341755 2333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:38:46.341740 2332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:38:46.342069 2327 server_base.cc:1034] running on GCE node
I20260927 02:38:46.342532 2327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:38:46.343101 2327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:38:46.344267 2327 hybrid_clock.cc:648] HybridClock initialized: now 1790476726344251 us; error 35 us; skew 500 ppm
Sep 27 02:38:46 dist-test-slave-zv8z krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:38:46.346994 2327 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:38:46.347786 2327 webserver.cc:460] Webserver started at http://127.0.98.2:44579/ using document root <none> and password file <none>
I20260927 02:38:46.348282 2327 fs_manager.cc:359] Metadata directory not provided
I20260927 02:38:46.348338 2327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:38:46.348498 2327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:38:46.350534 2327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-1/data/instance:
uuid: "7f69698959a44a8ea710e230956c873e"
format_stamp: "Formatted at 2026-09-27 02:38:46 on dist-test-slave-zv8z"
server_key: "b33fd30bde85c3ed17296c1f45badd47"
server_key_iv: "b1708da5ab7c67fb9d6818fce941d5d4"
server_key_version: "encryptionkey@0"
I20260927 02:38:46.351034 2327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7f69698959a44a8ea710e230956c873e"
format_stamp: "Formatted at 2026-09-27 02:38:46 on dist-test-slave-zv8z"
server_key: "b33fd30bde85c3ed17296c1f45badd47"
server_key_iv: "b1708da5ab7c67fb9d6818fce941d5d4"
server_key_version: "encryptionkey@0"
I20260927 02:38:46.354640 2327 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 02:38:46.357080 2342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:46.358333 2327 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:38:46.358435 2327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-1/wal
uuid: "7f69698959a44a8ea710e230956c873e"
format_stamp: "Formatted at 2026-09-27 02:38:46 on dist-test-slave-zv8z"
server_key: "b33fd30bde85c3ed17296c1f45badd47"
server_key_iv: "b1708da5ab7c67fb9d6818fce941d5d4"
server_key_version: "encryptionkey@0"
I20260927 02:38:46.358525 2327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:38:46.385181 2327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:38:46.388522 2327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:38:46.388695 2327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:38:46.390230 2327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:38:46.390316 2327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:46.390372 2327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:38:46.390401 2327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 02:38:46 dist-test-slave-zv8z krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:38:46.400254 2327 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45675
I20260927 02:38:46.400331 2457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45675 every 8 connection(s)
I20260927 02:38:46.401224 2327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:38:46.403340 2165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:46.391058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34479 (local address 127.0.98.62:33095)
0927 02:38:46.391209 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:46.391215 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:46.392079 (+ 864us) server_negotiation.cc:407] Connection header received
0927 02:38:46.393138 (+ 1059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:46.393142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:46.393202 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:46.393259 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:46.394760 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:46.395424 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:46.396633 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:46.396971 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:46.397081 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:46.399905 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:46.399909 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:46.399912 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:46.399940 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:46.401430 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:46.401985 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:46.401989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:46.401990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:46.402083 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:46.402444 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:46.402446 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:46.402447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:46.402626 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:46.402699 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:46.403135 (+ 436us) server_negotiation.cc:299] Negotiation successful
0927 02:38:46.403228 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 02:38:46.404151 2351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:46.391469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33095 (local address 127.0.98.2:34479)
0927 02:38:46.391924 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:46.391972 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:46.392896 (+ 924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:46.393673 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:46.393681 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:46.394038 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:38:46.394621 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:46.394635 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:46.395537 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:46.395541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:46.396490 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:46.396502 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:46.397147 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:46.397152 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:46.397376 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:46.398001 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:38:46.398024 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:38:46.399718 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:38:46.401536 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:46.401542 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:46.401557 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:46.401877 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:46.402173 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:46.402176 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:46.402178 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:46.402365 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:46.402703 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:38:46.402708 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:38:46.402945 (+ 237us) client_negotiation.cc:769] Sending connection context
0927 02:38:46.403092 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 02:38:46.403326 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":119,"threads_started":1}
I20260927 02:38:46.405267 2458 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33095
I20260927 02:38:46.405390 2458 heartbeater.cc:464] Registering TS with master...
I20260927 02:38:46.405823 2458 heartbeater.cc:511] Master 127.0.98.62:33095 requested a full tablet report, sending...
I20260927 02:38:46.406765 2115 ts_manager.cc:194] Registered new tserver with Master: 7f69698959a44a8ea710e230956c873e (127.0.98.2:45675)
I20260927 02:38:46.407388 2115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34479
I20260927 02:38:46.410782 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 2327
I20260927 02:38:46.410888 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:38:46.411176 392 external_mini_cluster.cc:1428] Setting key 9915f921f4afe9c73d0346356f90f76d
I20260927 02:38:46.413619 2165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:46.407250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38459 (local address 127.0.98.62:33095)
0927 02:38:46.407365 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:46.407370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:46.407488 (+ 118us) server_negotiation.cc:407] Connection header received
0927 02:38:46.407586 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:46.407588 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:46.407623 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:46.407689 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:46.408126 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:46.408616 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:46.409368 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:46.409588 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:46.409670 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:46.410695 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:46.410698 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:46.410700 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:46.410721 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:46.412107 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:46.412652 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:46.412655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:46.412656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:46.412725 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:46.412965 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:46.412967 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:46.412968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:46.413140 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:46.413208 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:46.413437 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 02:38:46.413517 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 02:38:46.413796 2464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:46.407167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33095 (local address 127.0.98.2:38459)
0927 02:38:46.407410 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:46.407424 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:46.407514 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:46.407720 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:46.407723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:46.407900 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:38:46.408021 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:46.408027 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:46.408728 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:46.408731 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:46.409270 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:46.409278 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:46.409677 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:46.409681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:46.409765 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:46.410186 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:38:46.410196 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:38:46.410571 (+ 375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:38:46.412276 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:46.412278 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:46.412279 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:46.412547 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:46.412799 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:46.412801 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:46.412803 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:46.412893 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:46.413212 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:38:46.413214 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:38:46.413296 (+ 82us) client_negotiation.cc:769] Sending connection context
0927 02:38:46.413400 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 02:38:46.413518 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":4,"thread_start_us":80,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:38:46.436713 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33095
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:38:46.542186 2467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:38:46.542455 2467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:38:46.542503 2467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:38:46.546383 2467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:38:46.546433 2467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:38:46.546469 2467 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 02:38:46.546481 2467 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:38:46.546515 2467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:38:46.551580 2467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:38:46.552511 2467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:38:46.553445 2467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:38:46.562500 2475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:38:46.562520 2472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:38:46.562520 2473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:38:46.562686 2467 server_base.cc:1034] running on GCE node
I20260927 02:38:46.563122 2467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:38:46.563714 2467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:38:46.564877 2467 hybrid_clock.cc:648] HybridClock initialized: now 1790476726564860 us; error 33 us; skew 500 ppm
Sep 27 02:38:46 dist-test-slave-zv8z krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:38:46.567619 2467 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:38:46.568638 2467 webserver.cc:460] Webserver started at http://127.0.98.3:33843/ using document root <none> and password file <none>
I20260927 02:38:46.569280 2467 fs_manager.cc:359] Metadata directory not provided
I20260927 02:38:46.569365 2467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:38:46.569569 2467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:38:46.571724 2467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-2/data/instance:
uuid: "a62ac2b710b342938b3a8b0b911d9c1e"
format_stamp: "Formatted at 2026-09-27 02:38:46 on dist-test-slave-zv8z"
server_key: "8d440beec0d8c098991e6a5256fdadc0"
server_key_iv: "1def5fbf2b22e9520d0629b6732644d9"
server_key_version: "encryptionkey@0"
I20260927 02:38:46.572242 2467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a62ac2b710b342938b3a8b0b911d9c1e"
format_stamp: "Formatted at 2026-09-27 02:38:46 on dist-test-slave-zv8z"
server_key: "8d440beec0d8c098991e6a5256fdadc0"
server_key_iv: "1def5fbf2b22e9520d0629b6732644d9"
server_key_version: "encryptionkey@0"
I20260927 02:38:46.575822 2467 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260927 02:38:46.578112 2482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:46.579291 2467 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:38:46.579380 2467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-2/wal
uuid: "a62ac2b710b342938b3a8b0b911d9c1e"
format_stamp: "Formatted at 2026-09-27 02:38:46 on dist-test-slave-zv8z"
server_key: "8d440beec0d8c098991e6a5256fdadc0"
server_key_iv: "1def5fbf2b22e9520d0629b6732644d9"
server_key_version: "encryptionkey@0"
I20260927 02:38:46.579512 2467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:38:46.594688 2467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:38:46.597664 2467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:38:46.597828 2467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:38:46.599223 2467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:38:46.599273 2467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:46.599324 2467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:38:46.599354 2467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 02:38:46 dist-test-slave-zv8z krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:38:46.609046 2467 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35547
I20260927 02:38:46.609133 2597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35547 every 8 connection(s)
I20260927 02:38:46.609925 2467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:38:46.612150 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 2467
I20260927 02:38:46.612257 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:38:46.612551 392 external_mini_cluster.cc:1428] Setting key a76e21c4eaf2eab2b33440787cd787ea
I20260927 02:38:46.613097 2165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:46.600052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40805 (local address 127.0.98.62:33095)
0927 02:38:46.600162 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:46.600166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:46.601185 (+ 1019us) server_negotiation.cc:407] Connection header received
0927 02:38:46.602222 (+ 1037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:46.602226 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:46.602380 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:46.602442 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:46.603916 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:46.604686 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:46.606359 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:46.606638 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:46.606719 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:46.609642 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:46.609646 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:46.609649 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:46.609674 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:46.611145 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:46.611641 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:46.611644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:46.611645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:46.611717 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:46.612050 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:46.612053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:46.612055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:46.612280 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:46.612373 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:46.612856 (+ 483us) server_negotiation.cc:299] Negotiation successful
0927 02:38:46.612954 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 02:38:46.613904 2491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:46.600580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33095 (local address 127.0.98.3:40805)
0927 02:38:46.601041 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:46.601082 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:46.601974 (+ 892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:46.602849 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:46.602856 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:46.603154 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:38:46.603779 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:46.603800 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:46.604823 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:46.604827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:46.606214 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:46.606226 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:46.606788 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:46.606792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:46.606998 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:46.607699 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:38:46.607719 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:38:46.609483 (+ 1764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:38:46.611257 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:46.611264 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:46.611278 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:46.611559 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:46.611826 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:46.611828 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:46.611830 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:46.611962 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:46.612387 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:38:46.612394 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:38:46.612628 (+ 234us) client_negotiation.cc:769] Sending connection context
0927 02:38:46.612815 (+ 187us) client_negotiation.cc:240] Negotiation successful
0927 02:38:46.613103 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":126,"threads_started":1}
I20260927 02:38:46.615010 2598 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33095
I20260927 02:38:46.615134 2598 heartbeater.cc:464] Registering TS with master...
I20260927 02:38:46.615567 2598 heartbeater.cc:511] Master 127.0.98.62:33095 requested a full tablet report, sending...
I20260927 02:38:46.616501 2115 ts_manager.cc:194] Registered new tserver with Master: a62ac2b710b342938b3a8b0b911d9c1e (127.0.98.3:35547)
I20260927 02:38:46.617117 2115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40805
I20260927 02:38:46.622852 2165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:46.616939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34687 (local address 127.0.98.62:33095)
0927 02:38:46.617047 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:46.617050 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:46.617159 (+ 109us) server_negotiation.cc:407] Connection header received
0927 02:38:46.617256 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:46.617258 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:46.617297 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:46.617367 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:46.617760 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:46.618278 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:46.619067 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:46.619280 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:46.619363 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:46.620274 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:46.620277 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:46.620280 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:46.620303 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:46.621408 (+ 1105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:46.621772 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:46.621775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:46.621777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:46.621868 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:46.622100 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:46.622103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:46.622105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:46.622299 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:46.622383 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:46.622617 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 02:38:46.622710 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 02:38:46.622862 2604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:46.616861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33095 (local address 127.0.98.3:34687)
0927 02:38:46.617087 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:46.617098 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:46.617185 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:46.617406 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:46.617409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:46.617568 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:38:46.617658 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:46.617662 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:46.618377 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:46.618380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:46.618965 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:46.618973 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:46.619373 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:46.619376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:46.619459 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:46.619879 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:38:46.619890 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:38:46.620154 (+ 264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:38:46.621505 (+ 1351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:46.621508 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:46.621510 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:46.621689 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:46.621947 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:46.621950 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:46.621952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:46.622026 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:46.622392 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:38:46.622395 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:38:46.622477 (+ 82us) client_negotiation.cc:769] Sending connection context
0927 02:38:46.622560 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 02:38:46.622686 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":74,"threads_started":1}
I20260927 02:38:46.626333 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 02:38:46 dist-test-slave-zv8z krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 02:38:46 dist-test-slave-zv8z krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:38:46.649179 2165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:46.641893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57180 (local address 127.0.98.62:33095)
0927 02:38:46.642020 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:46.642023 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:46.642034 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:38:46.642068 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:46.642071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:46.642105 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:46.642171 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:46.642830 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:46.643337 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:46.644128 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:46.644355 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:46.644443 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:46.646651 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:46.646654 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:46.646656 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:46.646680 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:46.647763 (+ 1083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:46.648126 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:46.648129 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:46.648130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:46.648202 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:46.648435 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:46.648439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:46.648440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:46.648605 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:46.648717 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:46.648931 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 02:38:46.649035 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 02:38:46.651780 2115 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57180:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:38:46.654897 2115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:38:46.668397 2615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:46.662679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43257 (local address 127.0.0.1:43668)
0927 02:38:46.663016 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:46.663070 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:46.663212 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:46.663936 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:46.663942 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:46.663968 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:38:46.664092 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:46.664099 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:46.665425 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:46.665428 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:46.666685 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:46.666694 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:46.667882 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:46.667886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:46.668002 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:46.668042 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 02:38:46.668134 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 02:38:46.668248 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":2,"thread_start_us":87,"threads_started":1}
I20260927 02:38:46.668546 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:46.663410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35547 (local address 127.0.0.1:58810)
0927 02:38:46.664189 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:46.664201 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:46.664343 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:46.664708 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:46.664711 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:46.664728 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:38:46.664811 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:46.664817 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:46.666057 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:46.666061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:46.667149 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:46.667157 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:46.668129 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:46.668132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:46.668226 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:46.668250 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 02:38:46.668303 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 02:38:46.668367 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"thread_start_us":93,"threads_started":1}
I20260927 02:38:46.668568 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:46.663615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58810 (local address 127.0.98.3:35547)
0927 02:38:46.664013 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:46.664024 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:46.664281 (+ 257us) server_negotiation.cc:407] Connection header received
0927 02:38:46.664439 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:46.664443 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:46.664586 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:46.664717 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:38:46.664898 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:46.665944 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:46.667258 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:46.668031 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:46.668134 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:46.668280 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:46.668343 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 02:38:46.668442 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":91,"threads_started":1}
I20260927 02:38:46.668802 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:46.662808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43668 (local address 127.0.98.1:43257)
0927 02:38:46.663073 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:46.663090 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:46.663145 (+ 55us) server_negotiation.cc:407] Connection header received
0927 02:38:46.663371 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:46.663376 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:46.663592 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:46.663750 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:38:46.664191 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:46.665310 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:46.666783 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:46.667567 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:46.667669 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:46.667822 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:46.668578 (+ 756us) server_negotiation.cc:299] Negotiation successful
0927 02:38:46.668673 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":83,"threads_started":1}
I20260927 02:38:46.671226 2532 tablet_service.cc:1467] Processing CreateTablet for tablet fa5b6883ed6d4de9a037942dd149bcba (DEFAULT_TABLE table=test-table [id=913aef4b5ea64c3b9522df5c58da12c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:38:46.671226 2252 tablet_service.cc:1467] Processing CreateTablet for tablet fa5b6883ed6d4de9a037942dd149bcba (DEFAULT_TABLE table=test-table [id=913aef4b5ea64c3b9522df5c58da12c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:38:46.672125 2252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa5b6883ed6d4de9a037942dd149bcba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:38:46.672135 2532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa5b6883ed6d4de9a037942dd149bcba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:38:46.678194 2621 tablet_bootstrap.cc:492] T fa5b6883ed6d4de9a037942dd149bcba P bfd99789611f4ab098a248d6366e52c9: Bootstrap starting.
I20260927 02:38:46.679425 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:46.663203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45675 (local address 127.0.0.1:41016)
0927 02:38:46.663945 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:46.663957 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:46.664062 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:46.664983 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:46.664986 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:46.665002 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:38:46.665081 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:46.665087 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:46.667250 (+ 2163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:46.667251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:46.667931 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:46.667937 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:46.679017 (+ 11080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:46.679021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:46.679134 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:46.679162 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 02:38:46.679214 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 02:38:46.679291 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"thread_start_us":90,"threads_started":1}
I20260927 02:38:46.679420 2622 tablet_bootstrap.cc:492] T fa5b6883ed6d4de9a037942dd149bcba P a62ac2b710b342938b3a8b0b911d9c1e: Bootstrap starting.
I20260927 02:38:46.680166 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:46.663581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41016 (local address 127.0.98.2:45675)
0927 02:38:46.664591 (+ 1010us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:46.664604 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:46.664624 (+ 20us) server_negotiation.cc:407] Connection header received
0927 02:38:46.664690 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:46.664695 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:46.664861 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:46.664999 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:38:46.665169 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:46.665948 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:46.668050 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:46.678873 (+ 10823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:46.679713 (+ 840us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:46.679883 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:46.679936 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 02:38:46.680028 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":913,"thread_start_us":111,"threads_started":1}
I20260927 02:38:46.680524 2621 tablet_bootstrap.cc:654] T fa5b6883ed6d4de9a037942dd149bcba P bfd99789611f4ab098a248d6366e52c9: Neither blocks nor log segments found. Creating new log.
I20260927 02:38:46.681142 2622 tablet_bootstrap.cc:654] T fa5b6883ed6d4de9a037942dd149bcba P a62ac2b710b342938b3a8b0b911d9c1e: Neither blocks nor log segments found. Creating new log.
I20260927 02:38:46.681324 2621 log.cc:824] T fa5b6883ed6d4de9a037942dd149bcba P bfd99789611f4ab098a248d6366e52c9: Log is configured to *not* fsync() on all Append() calls
I20260927 02:38:46.681936 2622 log.cc:824] T fa5b6883ed6d4de9a037942dd149bcba P a62ac2b710b342938b3a8b0b911d9c1e: Log is configured to *not* fsync() on all Append() calls
I20260927 02:38:46.683218 2621 tablet_bootstrap.cc:492] T fa5b6883ed6d4de9a037942dd149bcba P bfd99789611f4ab098a248d6366e52c9: No bootstrap required, opened a new log
I20260927 02:38:46.683406 2621 ts_tablet_manager.cc:1397] T fa5b6883ed6d4de9a037942dd149bcba P bfd99789611f4ab098a248d6366e52c9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 02:38:46.683022 2392 tablet_service.cc:1467] Processing CreateTablet for tablet fa5b6883ed6d4de9a037942dd149bcba (DEFAULT_TABLE table=test-table [id=913aef4b5ea64c3b9522df5c58da12c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:38:46.683876 2622 tablet_bootstrap.cc:492] T fa5b6883ed6d4de9a037942dd149bcba P a62ac2b710b342938b3a8b0b911d9c1e: No bootstrap required, opened a new log
I20260927 02:38:46.684067 2392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa5b6883ed6d4de9a037942dd149bcba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:38:46.684070 2622 ts_tablet_manager.cc:1397] T fa5b6883ed6d4de9a037942dd149bcba P a62ac2b710b342938b3a8b0b911d9c1e: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260927 02:38:46.686558 2621 raft_consensus.cc:348] T fa5b6883ed6d4de9a037942dd149bcba P bfd99789611f4ab098a248d6366e52c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } }
I20260927 02:38:46.686774 2621 raft_consensus.cc:374] T fa5b6883ed6d4de9a037942dd149bcba P bfd99789611f4ab098a248d6366e52c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:38:46.686832 2621 raft_consensus.cc:729] T fa5b6883ed6d4de9a037942dd149bcba P bfd99789611f4ab098a248d6366e52c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfd99789611f4ab098a248d6366e52c9, State: Initialized, Role: FOLLOWER
I20260927 02:38:46.687223 2621 consensus_queue.cc:260] T fa5b6883ed6d4de9a037942dd149bcba P bfd99789611f4ab098a248d6366e52c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } }
I20260927 02:38:46.687791 2318 heartbeater.cc:503] Master 127.0.98.62:33095 was elected leader, sending a full tablet report...
I20260927 02:38:46.688148 2622 raft_consensus.cc:348] T fa5b6883ed6d4de9a037942dd149bcba P a62ac2b710b342938b3a8b0b911d9c1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } }
I20260927 02:38:46.688385 2622 raft_consensus.cc:374] T fa5b6883ed6d4de9a037942dd149bcba P a62ac2b710b342938b3a8b0b911d9c1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:38:46.688448 2622 raft_consensus.cc:729] T fa5b6883ed6d4de9a037942dd149bcba P a62ac2b710b342938b3a8b0b911d9c1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a62ac2b710b342938b3a8b0b911d9c1e, State: Initialized, Role: FOLLOWER
I20260927 02:38:46.689006 2621 ts_tablet_manager.cc:1428] T fa5b6883ed6d4de9a037942dd149bcba P bfd99789611f4ab098a248d6366e52c9: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 02:38:46.689406 2622 consensus_queue.cc:260] T fa5b6883ed6d4de9a037942dd149bcba P a62ac2b710b342938b3a8b0b911d9c1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } }
I20260927 02:38:46.690332 2622 ts_tablet_manager.cc:1428] T fa5b6883ed6d4de9a037942dd149bcba P a62ac2b710b342938b3a8b0b911d9c1e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 02:38:46.690598 2598 heartbeater.cc:503] Master 127.0.98.62:33095 was elected leader, sending a full tablet report...
I20260927 02:38:46.692359 2627 tablet_bootstrap.cc:492] T fa5b6883ed6d4de9a037942dd149bcba P 7f69698959a44a8ea710e230956c873e: Bootstrap starting.
I20260927 02:38:46.694082 2627 tablet_bootstrap.cc:654] T fa5b6883ed6d4de9a037942dd149bcba P 7f69698959a44a8ea710e230956c873e: Neither blocks nor log segments found. Creating new log.
I20260927 02:38:46.694772 2627 log.cc:824] T fa5b6883ed6d4de9a037942dd149bcba P 7f69698959a44a8ea710e230956c873e: Log is configured to *not* fsync() on all Append() calls
I20260927 02:38:46.696341 2627 tablet_bootstrap.cc:492] T fa5b6883ed6d4de9a037942dd149bcba P 7f69698959a44a8ea710e230956c873e: No bootstrap required, opened a new log
I20260927 02:38:46.696511 2627 ts_tablet_manager.cc:1397] T fa5b6883ed6d4de9a037942dd149bcba P 7f69698959a44a8ea710e230956c873e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 02:38:46.699199 2627 raft_consensus.cc:348] T fa5b6883ed6d4de9a037942dd149bcba P 7f69698959a44a8ea710e230956c873e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } }
I20260927 02:38:46.699400 2627 raft_consensus.cc:374] T fa5b6883ed6d4de9a037942dd149bcba P 7f69698959a44a8ea710e230956c873e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:38:46.699465 2627 raft_consensus.cc:729] T fa5b6883ed6d4de9a037942dd149bcba P 7f69698959a44a8ea710e230956c873e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f69698959a44a8ea710e230956c873e, State: Initialized, Role: FOLLOWER
I20260927 02:38:46.699846 2627 consensus_queue.cc:260] T fa5b6883ed6d4de9a037942dd149bcba P 7f69698959a44a8ea710e230956c873e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } }
I20260927 02:38:46.700410 2458 heartbeater.cc:503] Master 127.0.98.62:33095 was elected leader, sending a full tablet report...
I20260927 02:38:46.700685 2627 ts_tablet_manager.cc:1428] T fa5b6883ed6d4de9a037942dd149bcba P 7f69698959a44a8ea710e230956c873e: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260927 02:38:46.730510 2625 raft_consensus.cc:482] T fa5b6883ed6d4de9a037942dd149bcba P bfd99789611f4ab098a248d6366e52c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:38:46.730737 2625 raft_consensus.cc:504] T fa5b6883ed6d4de9a037942dd149bcba P bfd99789611f4ab098a248d6366e52c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } }
I20260927 02:38:46.731911 2625 leader_election.cc:288] T fa5b6883ed6d4de9a037942dd149bcba P bfd99789611f4ab098a248d6366e52c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a62ac2b710b342938b3a8b0b911d9c1e (127.0.98.3:35547), 7f69698959a44a8ea710e230956c873e (127.0.98.2:45675)
I20260927 02:38:46.735647 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:46.732220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54055 (local address 127.0.98.3:35547)
0927 02:38:46.732345 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:46.732350 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:46.732539 (+ 189us) server_negotiation.cc:407] Connection header received
0927 02:38:46.732643 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:46.732646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:46.732701 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:46.732775 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:38:46.733055 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:46.733631 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:46.734716 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:46.735138 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:46.735214 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:46.735274 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:46.735453 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 02:38:46.735523 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 02:38:46.735662 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:46.732242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54339 (local address 127.0.98.2:45675)
0927 02:38:46.732345 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:46.732350 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:46.732721 (+ 371us) server_negotiation.cc:407] Connection header received
0927 02:38:46.732830 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:46.732833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:46.732881 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:46.732955 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:38:46.733178 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:46.733713 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:46.734783 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:46.735176 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:46.735274 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:46.735341 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:46.735485 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 02:38:46.735549 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 02:38:46.735677 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:46.732120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45675 (local address 127.0.98.1:54339)
0927 02:38:46.732638 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:46.732651 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:46.732751 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:46.732969 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:46.732971 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:46.732984 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:38:46.733095 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:46.733102 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:46.733811 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:46.733814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:46.734664 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:46.734675 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:46.735284 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:46.735288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:46.735370 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:46.735386 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 02:38:46.735465 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 02:38:46.735526 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"mutex_wait_us":234,"thread_start_us":81,"threads_started":1}
I20260927 02:38:46.735687 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:46.732113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35547 (local address 127.0.98.1:54055)
0927 02:38:46.732429 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:46.732445 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:46.732555 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:46.732797 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:46.732800 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:46.732825 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:38:46.732957 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:46.732964 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:46.733726 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:46.733729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:46.734605 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:46.734615 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:46.735246 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:46.735249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:46.735338 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:46.735361 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 02:38:46.735429 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 02:38:46.735496 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":19,"thread_start_us":96,"threads_started":1}
I20260927 02:38:46.736408 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa5b6883ed6d4de9a037942dd149bcba" candidate_uuid: "bfd99789611f4ab098a248d6366e52c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f69698959a44a8ea710e230956c873e" is_pre_election: true
I20260927 02:38:46.736409 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa5b6883ed6d4de9a037942dd149bcba" candidate_uuid: "bfd99789611f4ab098a248d6366e52c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" is_pre_election: true
I20260927 02:38:46.736652 2412 raft_consensus.cc:2454] T fa5b6883ed6d4de9a037942dd149bcba P 7f69698959a44a8ea710e230956c873e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfd99789611f4ab098a248d6366e52c9 in term 0.
I20260927 02:38:46.736663 2552 raft_consensus.cc:2454] T fa5b6883ed6d4de9a037942dd149bcba P a62ac2b710b342938b3a8b0b911d9c1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfd99789611f4ab098a248d6366e52c9 in term 0.
I20260927 02:38:46.737087 2206 leader_election.cc:302] T fa5b6883ed6d4de9a037942dd149bcba P bfd99789611f4ab098a248d6366e52c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7f69698959a44a8ea710e230956c873e, a62ac2b710b342938b3a8b0b911d9c1e, bfd99789611f4ab098a248d6366e52c9; no voters:
I20260927 02:38:46.737341 2625 raft_consensus.cc:2781] T fa5b6883ed6d4de9a037942dd149bcba P bfd99789611f4ab098a248d6366e52c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:38:46.737412 2625 raft_consensus.cc:482] T fa5b6883ed6d4de9a037942dd149bcba P bfd99789611f4ab098a248d6366e52c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:38:46.737453 2625 raft_consensus.cc:3037] T fa5b6883ed6d4de9a037942dd149bcba P bfd99789611f4ab098a248d6366e52c9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:38:46.738600 2625 raft_consensus.cc:504] T fa5b6883ed6d4de9a037942dd149bcba P bfd99789611f4ab098a248d6366e52c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } }
I20260927 02:38:46.738919 2625 leader_election.cc:288] T fa5b6883ed6d4de9a037942dd149bcba P bfd99789611f4ab098a248d6366e52c9 [CANDIDATE]: Term 1 election: Requested vote from peers a62ac2b710b342938b3a8b0b911d9c1e (127.0.98.3:35547), 7f69698959a44a8ea710e230956c873e (127.0.98.2:45675)
I20260927 02:38:46.739190 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa5b6883ed6d4de9a037942dd149bcba" candidate_uuid: "bfd99789611f4ab098a248d6366e52c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62ac2b710b342938b3a8b0b911d9c1e"
I20260927 02:38:46.739198 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa5b6883ed6d4de9a037942dd149bcba" candidate_uuid: "bfd99789611f4ab098a248d6366e52c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f69698959a44a8ea710e230956c873e"
I20260927 02:38:46.739303 2552 raft_consensus.cc:3037] T fa5b6883ed6d4de9a037942dd149bcba P a62ac2b710b342938b3a8b0b911d9c1e [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:38:46.739303 2412 raft_consensus.cc:3037] T fa5b6883ed6d4de9a037942dd149bcba P 7f69698959a44a8ea710e230956c873e [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:38:46.740574 2412 raft_consensus.cc:2454] T fa5b6883ed6d4de9a037942dd149bcba P 7f69698959a44a8ea710e230956c873e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfd99789611f4ab098a248d6366e52c9 in term 1.
I20260927 02:38:46.740574 2552 raft_consensus.cc:2454] T fa5b6883ed6d4de9a037942dd149bcba P a62ac2b710b342938b3a8b0b911d9c1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfd99789611f4ab098a248d6366e52c9 in term 1.
I20260927 02:38:46.740855 2203 leader_election.cc:302] T fa5b6883ed6d4de9a037942dd149bcba P bfd99789611f4ab098a248d6366e52c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a62ac2b710b342938b3a8b0b911d9c1e, bfd99789611f4ab098a248d6366e52c9; no voters:
I20260927 02:38:46.741003 2625 raft_consensus.cc:2781] T fa5b6883ed6d4de9a037942dd149bcba P bfd99789611f4ab098a248d6366e52c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:38:46.741166 2625 raft_consensus.cc:686] T fa5b6883ed6d4de9a037942dd149bcba P bfd99789611f4ab098a248d6366e52c9 [term 1 LEADER]: Becoming Leader. State: Replica: bfd99789611f4ab098a248d6366e52c9, State: Running, Role: LEADER
I20260927 02:38:46.741446 2625 consensus_queue.cc:237] T fa5b6883ed6d4de9a037942dd149bcba P bfd99789611f4ab098a248d6366e52c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } }
I20260927 02:38:46.744287 2114 catalog_manager.cc:5495] T fa5b6883ed6d4de9a037942dd149bcba P bfd99789611f4ab098a248d6366e52c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to bfd99789611f4ab098a248d6366e52c9 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "bfd99789611f4ab098a248d6366e52c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } health_report { overall_health: UNKNOWN } } }
Sep 27 02:38:46 dist-test-slave-zv8z krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:38:46.792063 2165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:46.783977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57188 (local address 127.0.98.62:33095)
0927 02:38:46.784093 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:46.784097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:46.784278 (+ 181us) server_negotiation.cc:407] Connection header received
0927 02:38:46.784394 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:46.784397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:46.784434 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:46.784499 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:46.785145 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:46.785646 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:46.786401 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:46.786606 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:46.786689 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:46.788791 (+ 2102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:46.788795 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:46.788797 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:46.788816 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:46.790191 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:46.790832 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:46.790835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:46.790837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:46.790923 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:46.791162 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:46.791164 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:46.791166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:46.791344 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:46.791502 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:46.791770 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 02:38:46.791914 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":39}
I20260927 02:38:46.792073 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:46.783913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33095 (local address 127.0.0.1:57188)
0927 02:38:46.784197 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:46.784208 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:46.784313 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:46.784554 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:46.784557 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:46.784941 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:38:46.785036 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:46.785042 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:46.785737 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:46.785740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:46.786302 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:46.786310 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:46.786695 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:46.786699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:46.786803 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:46.787299 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:38:46.787310 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:38:46.788674 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:38:46.790306 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:46.790315 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:46.790321 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:46.790748 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:46.791003 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:46.791006 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:46.791008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:46.791086 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:46.791505 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:38:46.791512 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:38:46.791622 (+ 110us) client_negotiation.cc:769] Sending connection context
0927 02:38:46.791717 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 02:38:46.791865 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":17,"thread_start_us":121,"threads_started":1}
I20260927 02:38:46.795773 2114 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57188:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 02:38:46.796417 2114 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:38:46.800721 2252 tablet_service.cc:1467] Processing CreateTablet for tablet 00af8318c4744a1a86fb650153773ae6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=09215df117b24231bf39e25e462cd1ef]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:38:46.800928 2392 tablet_service.cc:1467] Processing CreateTablet for tablet 00af8318c4744a1a86fb650153773ae6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=09215df117b24231bf39e25e462cd1ef]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:38:46.801499 2252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00af8318c4744a1a86fb650153773ae6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:38:46.801499 2392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00af8318c4744a1a86fb650153773ae6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:38:46.801180 2532 tablet_service.cc:1467] Processing CreateTablet for tablet 00af8318c4744a1a86fb650153773ae6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=09215df117b24231bf39e25e462cd1ef]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:38:46.801853 2532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00af8318c4744a1a86fb650153773ae6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:38:46.804293 2627 tablet_bootstrap.cc:492] T 00af8318c4744a1a86fb650153773ae6 P 7f69698959a44a8ea710e230956c873e: Bootstrap starting.
I20260927 02:38:46.804301 2622 tablet_bootstrap.cc:492] T 00af8318c4744a1a86fb650153773ae6 P a62ac2b710b342938b3a8b0b911d9c1e: Bootstrap starting.
I20260927 02:38:46.804301 2621 tablet_bootstrap.cc:492] T 00af8318c4744a1a86fb650153773ae6 P bfd99789611f4ab098a248d6366e52c9: Bootstrap starting.
I20260927 02:38:46.805119 2627 tablet_bootstrap.cc:654] T 00af8318c4744a1a86fb650153773ae6 P 7f69698959a44a8ea710e230956c873e: Neither blocks nor log segments found. Creating new log.
I20260927 02:38:46.805287 2622 tablet_bootstrap.cc:654] T 00af8318c4744a1a86fb650153773ae6 P a62ac2b710b342938b3a8b0b911d9c1e: Neither blocks nor log segments found. Creating new log.
I20260927 02:38:46.805595 2621 tablet_bootstrap.cc:654] T 00af8318c4744a1a86fb650153773ae6 P bfd99789611f4ab098a248d6366e52c9: Neither blocks nor log segments found. Creating new log.
I20260927 02:38:46.806216 2627 tablet_bootstrap.cc:492] T 00af8318c4744a1a86fb650153773ae6 P 7f69698959a44a8ea710e230956c873e: No bootstrap required, opened a new log
I20260927 02:38:46.806313 2627 ts_tablet_manager.cc:1397] T 00af8318c4744a1a86fb650153773ae6 P 7f69698959a44a8ea710e230956c873e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:38:46.806759 2621 tablet_bootstrap.cc:492] T 00af8318c4744a1a86fb650153773ae6 P bfd99789611f4ab098a248d6366e52c9: No bootstrap required, opened a new log
I20260927 02:38:46.806743 2627 raft_consensus.cc:348] T 00af8318c4744a1a86fb650153773ae6 P 7f69698959a44a8ea710e230956c873e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } }
I20260927 02:38:46.806828 2621 ts_tablet_manager.cc:1397] T 00af8318c4744a1a86fb650153773ae6 P bfd99789611f4ab098a248d6366e52c9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 02:38:46.806835 2627 raft_consensus.cc:374] T 00af8318c4744a1a86fb650153773ae6 P 7f69698959a44a8ea710e230956c873e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:38:46.806865 2627 raft_consensus.cc:729] T 00af8318c4744a1a86fb650153773ae6 P 7f69698959a44a8ea710e230956c873e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f69698959a44a8ea710e230956c873e, State: Initialized, Role: FOLLOWER
I20260927 02:38:46.806974 2627 consensus_queue.cc:260] T 00af8318c4744a1a86fb650153773ae6 P 7f69698959a44a8ea710e230956c873e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } }
I20260927 02:38:46.807219 2627 ts_tablet_manager.cc:1428] T 00af8318c4744a1a86fb650153773ae6 P 7f69698959a44a8ea710e230956c873e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 02:38:46.807217 2629 tablet_replica.cc:439] T 00af8318c4744a1a86fb650153773ae6 P 7f69698959a44a8ea710e230956c873e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } } }
I20260927 02:38:46.807252 2621 raft_consensus.cc:348] T 00af8318c4744a1a86fb650153773ae6 P bfd99789611f4ab098a248d6366e52c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } }
I20260927 02:38:46.807348 2629 tablet_replica.cc:442] T 00af8318c4744a1a86fb650153773ae6 P 7f69698959a44a8ea710e230956c873e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:38:46.807350 2621 raft_consensus.cc:374] T 00af8318c4744a1a86fb650153773ae6 P bfd99789611f4ab098a248d6366e52c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:38:46.807380 2621 raft_consensus.cc:729] T 00af8318c4744a1a86fb650153773ae6 P bfd99789611f4ab098a248d6366e52c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfd99789611f4ab098a248d6366e52c9, State: Initialized, Role: FOLLOWER
I20260927 02:38:46.807499 2621 consensus_queue.cc:260] T 00af8318c4744a1a86fb650153773ae6 P bfd99789611f4ab098a248d6366e52c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } }
I20260927 02:38:46.807750 2621 ts_tablet_manager.cc:1428] T 00af8318c4744a1a86fb650153773ae6 P bfd99789611f4ab098a248d6366e52c9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:38:46.807745 2625 tablet_replica.cc:439] T 00af8318c4744a1a86fb650153773ae6 P bfd99789611f4ab098a248d6366e52c9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } } }
I20260927 02:38:46.807868 2625 tablet_replica.cc:442] T 00af8318c4744a1a86fb650153773ae6 P bfd99789611f4ab098a248d6366e52c9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:38:46.807950 2622 tablet_bootstrap.cc:492] T 00af8318c4744a1a86fb650153773ae6 P a62ac2b710b342938b3a8b0b911d9c1e: No bootstrap required, opened a new log
I20260927 02:38:46.808038 2622 ts_tablet_manager.cc:1397] T 00af8318c4744a1a86fb650153773ae6 P a62ac2b710b342938b3a8b0b911d9c1e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 02:38:46.808429 2622 raft_consensus.cc:348] T 00af8318c4744a1a86fb650153773ae6 P a62ac2b710b342938b3a8b0b911d9c1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } }
I20260927 02:38:46.808525 2622 raft_consensus.cc:374] T 00af8318c4744a1a86fb650153773ae6 P a62ac2b710b342938b3a8b0b911d9c1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:38:46.808557 2622 raft_consensus.cc:729] T 00af8318c4744a1a86fb650153773ae6 P a62ac2b710b342938b3a8b0b911d9c1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a62ac2b710b342938b3a8b0b911d9c1e, State: Initialized, Role: FOLLOWER
I20260927 02:38:46.808666 2622 consensus_queue.cc:260] T 00af8318c4744a1a86fb650153773ae6 P a62ac2b710b342938b3a8b0b911d9c1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } }
I20260927 02:38:46.808919 2622 ts_tablet_manager.cc:1428] T 00af8318c4744a1a86fb650153773ae6 P a62ac2b710b342938b3a8b0b911d9c1e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 02:38:46.808933 2626 tablet_replica.cc:439] T 00af8318c4744a1a86fb650153773ae6 P a62ac2b710b342938b3a8b0b911d9c1e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } } }
I20260927 02:38:46.809075 2626 tablet_replica.cc:442] T 00af8318c4744a1a86fb650153773ae6 P a62ac2b710b342938b3a8b0b911d9c1e: This TxnStatusTablet replica's current role is: FOLLOWER
W20260927 02:38:46.861126 2599 tablet.cc:2369] T 00af8318c4744a1a86fb650153773ae6 P a62ac2b710b342938b3a8b0b911d9c1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:38:46.902393 2459 tablet.cc:2369] T 00af8318c4744a1a86fb650153773ae6 P 7f69698959a44a8ea710e230956c873e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:38:46.936059 2319 tablet.cc:2369] T 00af8318c4744a1a86fb650153773ae6 P bfd99789611f4ab098a248d6366e52c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:38:47.206909 2625 raft_consensus.cc:482] T 00af8318c4744a1a86fb650153773ae6 P bfd99789611f4ab098a248d6366e52c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:38:47.207047 2625 raft_consensus.cc:504] T 00af8318c4744a1a86fb650153773ae6 P bfd99789611f4ab098a248d6366e52c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } }
I20260927 02:38:47.207410 2625 leader_election.cc:288] T 00af8318c4744a1a86fb650153773ae6 P bfd99789611f4ab098a248d6366e52c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f69698959a44a8ea710e230956c873e (127.0.98.2:45675), a62ac2b710b342938b3a8b0b911d9c1e (127.0.98.3:35547)
I20260927 02:38:47.207886 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00af8318c4744a1a86fb650153773ae6" candidate_uuid: "bfd99789611f4ab098a248d6366e52c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f69698959a44a8ea710e230956c873e" is_pre_election: true
I20260927 02:38:47.207892 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00af8318c4744a1a86fb650153773ae6" candidate_uuid: "bfd99789611f4ab098a248d6366e52c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" is_pre_election: true
I20260927 02:38:47.208029 2412 raft_consensus.cc:2454] T 00af8318c4744a1a86fb650153773ae6 P 7f69698959a44a8ea710e230956c873e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfd99789611f4ab098a248d6366e52c9 in term 0.
I20260927 02:38:47.208036 2552 raft_consensus.cc:2454] T 00af8318c4744a1a86fb650153773ae6 P a62ac2b710b342938b3a8b0b911d9c1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfd99789611f4ab098a248d6366e52c9 in term 0.
I20260927 02:38:47.208326 2206 leader_election.cc:302] T 00af8318c4744a1a86fb650153773ae6 P bfd99789611f4ab098a248d6366e52c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7f69698959a44a8ea710e230956c873e, a62ac2b710b342938b3a8b0b911d9c1e, bfd99789611f4ab098a248d6366e52c9; no voters:
I20260927 02:38:47.208534 2625 raft_consensus.cc:2781] T 00af8318c4744a1a86fb650153773ae6 P bfd99789611f4ab098a248d6366e52c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:38:47.208595 2625 raft_consensus.cc:482] T 00af8318c4744a1a86fb650153773ae6 P bfd99789611f4ab098a248d6366e52c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:38:47.208640 2625 raft_consensus.cc:3037] T 00af8318c4744a1a86fb650153773ae6 P bfd99789611f4ab098a248d6366e52c9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:38:47.209547 2625 raft_consensus.cc:504] T 00af8318c4744a1a86fb650153773ae6 P bfd99789611f4ab098a248d6366e52c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } }
I20260927 02:38:47.209856 2625 leader_election.cc:288] T 00af8318c4744a1a86fb650153773ae6 P bfd99789611f4ab098a248d6366e52c9 [CANDIDATE]: Term 1 election: Requested vote from peers 7f69698959a44a8ea710e230956c873e (127.0.98.2:45675), a62ac2b710b342938b3a8b0b911d9c1e (127.0.98.3:35547)
I20260927 02:38:47.210142 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00af8318c4744a1a86fb650153773ae6" candidate_uuid: "bfd99789611f4ab098a248d6366e52c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f69698959a44a8ea710e230956c873e"
I20260927 02:38:47.210187 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00af8318c4744a1a86fb650153773ae6" candidate_uuid: "bfd99789611f4ab098a248d6366e52c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62ac2b710b342938b3a8b0b911d9c1e"
I20260927 02:38:47.210286 2412 raft_consensus.cc:3037] T 00af8318c4744a1a86fb650153773ae6 P 7f69698959a44a8ea710e230956c873e [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:38:47.210325 2552 raft_consensus.cc:3037] T 00af8318c4744a1a86fb650153773ae6 P a62ac2b710b342938b3a8b0b911d9c1e [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:38:47.211333 2412 raft_consensus.cc:2454] T 00af8318c4744a1a86fb650153773ae6 P 7f69698959a44a8ea710e230956c873e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfd99789611f4ab098a248d6366e52c9 in term 1.
I20260927 02:38:47.211331 2552 raft_consensus.cc:2454] T 00af8318c4744a1a86fb650153773ae6 P a62ac2b710b342938b3a8b0b911d9c1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfd99789611f4ab098a248d6366e52c9 in term 1.
I20260927 02:38:47.211627 2206 leader_election.cc:302] T 00af8318c4744a1a86fb650153773ae6 P bfd99789611f4ab098a248d6366e52c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f69698959a44a8ea710e230956c873e, bfd99789611f4ab098a248d6366e52c9; no voters:
I20260927 02:38:47.211800 2625 raft_consensus.cc:2781] T 00af8318c4744a1a86fb650153773ae6 P bfd99789611f4ab098a248d6366e52c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:38:47.211885 2625 raft_consensus.cc:686] T 00af8318c4744a1a86fb650153773ae6 P bfd99789611f4ab098a248d6366e52c9 [term 1 LEADER]: Becoming Leader. State: Replica: bfd99789611f4ab098a248d6366e52c9, State: Running, Role: LEADER
I20260927 02:38:47.212021 2625 consensus_queue.cc:237] T 00af8318c4744a1a86fb650153773ae6 P bfd99789611f4ab098a248d6366e52c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } }
I20260927 02:38:47.212617 2632 tablet_replica.cc:439] T 00af8318c4744a1a86fb650153773ae6 P bfd99789611f4ab098a248d6366e52c9: TxnStatusTablet state changed. Reason: New leader bfd99789611f4ab098a248d6366e52c9. Latest consensus state: current_term: 1 leader_uuid: "bfd99789611f4ab098a248d6366e52c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } } }
I20260927 02:38:47.212723 2632 tablet_replica.cc:442] T 00af8318c4744a1a86fb650153773ae6 P bfd99789611f4ab098a248d6366e52c9: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:38:47.212975 2644 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 02:38:47.213203 2644 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 02:38:47.213621 2114 catalog_manager.cc:5495] T 00af8318c4744a1a86fb650153773ae6 P bfd99789611f4ab098a248d6366e52c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to bfd99789611f4ab098a248d6366e52c9 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "bfd99789611f4ab098a248d6366e52c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } health_report { overall_health: UNKNOWN } } }
I20260927 02:38:47.256690 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:47.253108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43676 (local address 127.0.98.1:43257)
0927 02:38:47.253322 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:47.253326 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:47.253339 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:38:47.253416 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:47.253419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:47.253486 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:47.253578 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:38:47.253816 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:47.254391 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:47.255236 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:47.255482 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:47.255569 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:47.255871 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:38:47.256282 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:38:47.256396 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:47.256476 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 02:38:47.256528 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":61,"threads_started":1}
I20260927 02:38:47.256670 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:47.253025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43257 (local address 127.0.0.1:43676)
0927 02:38:47.253167 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:47.253183 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:47.253294 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:47.253608 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:47.253611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:47.253619 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:38:47.253719 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:47.253725 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:47.254498 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:47.254501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:47.255120 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:47.255128 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:47.255584 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:47.255588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:47.255690 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:47.255731 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:38:47.256388 (+ 657us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:38:47.256402 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 02:38:47.256466 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 02:38:47.256551 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 02:38:47.262044 2552 raft_consensus.cc:1261] T 00af8318c4744a1a86fb650153773ae6 P a62ac2b710b342938b3a8b0b911d9c1e [term 1 FOLLOWER]: Refusing update from remote peer bfd99789611f4ab098a248d6366e52c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:38:47.262130 2412 raft_consensus.cc:1261] T 00af8318c4744a1a86fb650153773ae6 P 7f69698959a44a8ea710e230956c873e [term 1 FOLLOWER]: Refusing update from remote peer bfd99789611f4ab098a248d6366e52c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:38:47.262751 2632 consensus_queue.cc:1035] T 00af8318c4744a1a86fb650153773ae6 P bfd99789611f4ab098a248d6366e52c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:38:47.262934 2625 consensus_queue.cc:1035] T 00af8318c4744a1a86fb650153773ae6 P bfd99789611f4ab098a248d6366e52c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:38:47.266171 2626 tablet_replica.cc:439] T 00af8318c4744a1a86fb650153773ae6 P a62ac2b710b342938b3a8b0b911d9c1e: TxnStatusTablet state changed. Reason: New leader bfd99789611f4ab098a248d6366e52c9. Latest consensus state: current_term: 1 leader_uuid: "bfd99789611f4ab098a248d6366e52c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } } }
I20260927 02:38:47.266314 2626 tablet_replica.cc:442] T 00af8318c4744a1a86fb650153773ae6 P a62ac2b710b342938b3a8b0b911d9c1e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:38:47.266574 2629 tablet_replica.cc:439] T 00af8318c4744a1a86fb650153773ae6 P 7f69698959a44a8ea710e230956c873e: TxnStatusTablet state changed. Reason: New leader bfd99789611f4ab098a248d6366e52c9. Latest consensus state: current_term: 1 leader_uuid: "bfd99789611f4ab098a248d6366e52c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } } }
I20260927 02:38:47.266683 2629 tablet_replica.cc:442] T 00af8318c4744a1a86fb650153773ae6 P 7f69698959a44a8ea710e230956c873e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:38:47.269676 2632 tablet_replica.cc:439] T 00af8318c4744a1a86fb650153773ae6 P bfd99789611f4ab098a248d6366e52c9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bfd99789611f4ab098a248d6366e52c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } } }
I20260927 02:38:47.269783 2632 tablet_replica.cc:442] T 00af8318c4744a1a86fb650153773ae6 P bfd99789611f4ab098a248d6366e52c9: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:38:47.269870 2648 mvcc.cc:205] Tried to move back new op lower bound from 7333792674858934272 to 7333792674661687296. Current Snapshot: MvccSnapshot[applied={T|T < 7333792674858934272}]
I20260927 02:38:47.270313 2629 tablet_replica.cc:439] T 00af8318c4744a1a86fb650153773ae6 P 7f69698959a44a8ea710e230956c873e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bfd99789611f4ab098a248d6366e52c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } } }
I20260927 02:38:47.270416 2629 tablet_replica.cc:442] T 00af8318c4744a1a86fb650153773ae6 P 7f69698959a44a8ea710e230956c873e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:38:47.275913 2632 consensus_queue.cc:1035] T fa5b6883ed6d4de9a037942dd149bcba P bfd99789611f4ab098a248d6366e52c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:38:47.276415 2632 tablet_replica.cc:439] T 00af8318c4744a1a86fb650153773ae6 P bfd99789611f4ab098a248d6366e52c9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bfd99789611f4ab098a248d6366e52c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } } }
I20260927 02:38:47.276528 2632 tablet_replica.cc:442] T 00af8318c4744a1a86fb650153773ae6 P bfd99789611f4ab098a248d6366e52c9: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:38:47.277707 2626 tablet_replica.cc:439] T 00af8318c4744a1a86fb650153773ae6 P a62ac2b710b342938b3a8b0b911d9c1e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bfd99789611f4ab098a248d6366e52c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } } }
I20260927 02:38:47.277820 2626 tablet_replica.cc:442] T 00af8318c4744a1a86fb650153773ae6 P a62ac2b710b342938b3a8b0b911d9c1e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:38:47.279814 2658 consensus_queue.cc:1035] T fa5b6883ed6d4de9a037942dd149bcba P bfd99789611f4ab098a248d6366e52c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:38:47.284063 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:47.280407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43680 (local address 127.0.98.1:43257)
0927 02:38:47.280517 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:47.280521 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:47.280762 (+ 241us) server_negotiation.cc:407] Connection header received
0927 02:38:47.280964 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:47.280967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:47.281018 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:47.281067 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:38:47.281515 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:47.282093 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:47.283083 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:47.283330 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:47.283415 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:47.283674 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:38:47.283743 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:38:47.283800 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:47.283888 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 02:38:47.283939 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 02:38:47.284830 2650 mvcc.cc:205] Tried to move back new op lower bound from 7333792674858934272 to 7333792674661687296. Current Snapshot: MvccSnapshot[applied={T|T < 7333792674858934272 or (T in {7333792674858934272})}]
I20260927 02:38:47.294184 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:47.291132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46573 (local address 127.0.98.1:43257)
0927 02:38:47.291228 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:47.291232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:47.291402 (+ 170us) server_negotiation.cc:407] Connection header received
0927 02:38:47.291589 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:47.291592 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:47.291631 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:47.291685 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:38:47.291895 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:47.292420 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:47.293244 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:47.293457 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:47.293537 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:47.293813 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:38:47.293876 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:38:47.293938 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:47.294010 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 02:38:47.294049 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 02:38:47.294196 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:47.291087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43257 (local address 127.0.98.1:46573)
0927 02:38:47.291332 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:47.291344 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:47.291489 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:47.291712 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:47.291715 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:47.291722 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:38:47.291815 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:47.291819 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:47.292504 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:47.292507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:47.293135 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:47.293143 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:47.293564 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:47.293567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:47.293651 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:47.293685 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:38:47.293942 (+ 257us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:38:47.293947 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:38:47.294003 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 02:38:47.294073 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":1,"thread_start_us":93,"threads_started":1}
I20260927 02:38:47.342571 2113 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:38:47.342775 2113 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:38:47.345361 2113 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 336ef4719fc24eb4aeb74a965882da30: Sending DeleteTablet for 3 replicas of tablet fa5b6883ed6d4de9a037942dd149bcba
I20260927 02:38:47.346225 2532 tablet_service.cc:1514] Processing DeleteTablet for tablet fa5b6883ed6d4de9a037942dd149bcba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:38:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58810
I20260927 02:38:47.346727 2392 tablet_service.cc:1514] Processing DeleteTablet for tablet fa5b6883ed6d4de9a037942dd149bcba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:38:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41016
I20260927 02:38:47.346807 2251 tablet_service.cc:1514] Processing DeleteTablet for tablet fa5b6883ed6d4de9a037942dd149bcba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:38:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43668
I20260927 02:38:47.346835 2113 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57180:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 02:38:47.347183 2668 tablet_replica.cc:330] T fa5b6883ed6d4de9a037942dd149bcba P a62ac2b710b342938b3a8b0b911d9c1e: stopping tablet replica
I20260927 02:38:47.347175 2669 tablet_replica.cc:330] T fa5b6883ed6d4de9a037942dd149bcba P 7f69698959a44a8ea710e230956c873e: stopping tablet replica
W20260927 02:38:47.347254 2113 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:38:47.347486 2668 raft_consensus.cc:2229] T fa5b6883ed6d4de9a037942dd149bcba P a62ac2b710b342938b3a8b0b911d9c1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:38:47.347486 2669 raft_consensus.cc:2229] T fa5b6883ed6d4de9a037942dd149bcba P 7f69698959a44a8ea710e230956c873e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:38:47.347688 2668 raft_consensus.cc:2258] T fa5b6883ed6d4de9a037942dd149bcba P a62ac2b710b342938b3a8b0b911d9c1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:38:47.347688 2669 raft_consensus.cc:2258] T fa5b6883ed6d4de9a037942dd149bcba P 7f69698959a44a8ea710e230956c873e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:38:47.348045 2670 tablet_replica.cc:330] T fa5b6883ed6d4de9a037942dd149bcba P bfd99789611f4ab098a248d6366e52c9: stopping tablet replica
I20260927 02:38:47.348315 2670 raft_consensus.cc:2229] T fa5b6883ed6d4de9a037942dd149bcba P bfd99789611f4ab098a248d6366e52c9 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:38:47.348657 2670 raft_consensus.cc:2258] T fa5b6883ed6d4de9a037942dd149bcba P bfd99789611f4ab098a248d6366e52c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:38:47.349228 2668 ts_tablet_manager.cc:1905] T fa5b6883ed6d4de9a037942dd149bcba P a62ac2b710b342938b3a8b0b911d9c1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:38:47.350375 2670 ts_tablet_manager.cc:1905] T fa5b6883ed6d4de9a037942dd149bcba P bfd99789611f4ab098a248d6366e52c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:38:47.351130 2669 ts_tablet_manager.cc:1905] T fa5b6883ed6d4de9a037942dd149bcba P 7f69698959a44a8ea710e230956c873e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:38:47.352340 2251 tablet_service.cc:1467] Processing CreateTablet for tablet 95654a75a4954c9480568d5077d059bc (DEFAULT_TABLE table=test-table [id=32602e0bab16478cbd024d3122415623]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:38:47.352391 2532 tablet_service.cc:1467] Processing CreateTablet for tablet 95654a75a4954c9480568d5077d059bc (DEFAULT_TABLE table=test-table [id=32602e0bab16478cbd024d3122415623]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:38:47.352599 2251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95654a75a4954c9480568d5077d059bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:38:47.352623 2532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95654a75a4954c9480568d5077d059bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:38:47.353833 2669 ts_tablet_manager.cc:1918] T fa5b6883ed6d4de9a037942dd149bcba P 7f69698959a44a8ea710e230956c873e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:38:47.353907 2669 log.cc:1196] T fa5b6883ed6d4de9a037942dd149bcba P 7f69698959a44a8ea710e230956c873e: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-1/wal/wals/fa5b6883ed6d4de9a037942dd149bcba
I20260927 02:38:47.354182 2669 ts_tablet_manager.cc:1939] T fa5b6883ed6d4de9a037942dd149bcba P 7f69698959a44a8ea710e230956c873e: Deleting consensus metadata
I20260927 02:38:47.354758 2392 tablet_service.cc:1467] Processing CreateTablet for tablet 95654a75a4954c9480568d5077d059bc (DEFAULT_TABLE table=test-table [id=32602e0bab16478cbd024d3122415623]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:38:47.354978 2392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95654a75a4954c9480568d5077d059bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:38:47.355376 2102 catalog_manager.cc:4841] TS 7f69698959a44a8ea710e230956c873e (127.0.98.2:45675): tablet fa5b6883ed6d4de9a037942dd149bcba (table test-table [id=913aef4b5ea64c3b9522df5c58da12c0]) successfully deleted
I20260927 02:38:47.357096 2672 tablet_bootstrap.cc:492] T 95654a75a4954c9480568d5077d059bc P 7f69698959a44a8ea710e230956c873e: Bootstrap starting.
I20260927 02:38:47.357581 2668 ts_tablet_manager.cc:1918] T fa5b6883ed6d4de9a037942dd149bcba P a62ac2b710b342938b3a8b0b911d9c1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:38:47.357673 2668 log.cc:1196] T fa5b6883ed6d4de9a037942dd149bcba P a62ac2b710b342938b3a8b0b911d9c1e: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-2/wal/wals/fa5b6883ed6d4de9a037942dd149bcba
I20260927 02:38:47.357970 2668 ts_tablet_manager.cc:1939] T fa5b6883ed6d4de9a037942dd149bcba P a62ac2b710b342938b3a8b0b911d9c1e: Deleting consensus metadata
I20260927 02:38:47.358186 2672 tablet_bootstrap.cc:654] T 95654a75a4954c9480568d5077d059bc P 7f69698959a44a8ea710e230956c873e: Neither blocks nor log segments found. Creating new log.
I20260927 02:38:47.359120 2099 catalog_manager.cc:4841] TS a62ac2b710b342938b3a8b0b911d9c1e (127.0.98.3:35547): tablet fa5b6883ed6d4de9a037942dd149bcba (table test-table [id=913aef4b5ea64c3b9522df5c58da12c0]) successfully deleted
I20260927 02:38:47.359308 2672 tablet_bootstrap.cc:492] T 95654a75a4954c9480568d5077d059bc P 7f69698959a44a8ea710e230956c873e: No bootstrap required, opened a new log
I20260927 02:38:47.359375 2672 ts_tablet_manager.cc:1397] T 95654a75a4954c9480568d5077d059bc P 7f69698959a44a8ea710e230956c873e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260927 02:38:47.359795 2672 raft_consensus.cc:348] T 95654a75a4954c9480568d5077d059bc P 7f69698959a44a8ea710e230956c873e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } }
I20260927 02:38:47.359896 2672 raft_consensus.cc:374] T 95654a75a4954c9480568d5077d059bc P 7f69698959a44a8ea710e230956c873e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:38:47.359922 2672 raft_consensus.cc:729] T 95654a75a4954c9480568d5077d059bc P 7f69698959a44a8ea710e230956c873e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f69698959a44a8ea710e230956c873e, State: Initialized, Role: FOLLOWER
I20260927 02:38:47.360044 2672 consensus_queue.cc:260] T 95654a75a4954c9480568d5077d059bc P 7f69698959a44a8ea710e230956c873e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } }
I20260927 02:38:47.360297 2672 ts_tablet_manager.cc:1428] T 95654a75a4954c9480568d5077d059bc P 7f69698959a44a8ea710e230956c873e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 02:38:47.361356 2670 ts_tablet_manager.cc:1918] T fa5b6883ed6d4de9a037942dd149bcba P bfd99789611f4ab098a248d6366e52c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:38:47.361450 2670 log.cc:1196] T fa5b6883ed6d4de9a037942dd149bcba P bfd99789611f4ab098a248d6366e52c9: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-0/wal/wals/fa5b6883ed6d4de9a037942dd149bcba
I20260927 02:38:47.361742 2670 ts_tablet_manager.cc:1939] T fa5b6883ed6d4de9a037942dd149bcba P bfd99789611f4ab098a248d6366e52c9: Deleting consensus metadata
I20260927 02:38:47.362877 2099 catalog_manager.cc:4841] TS bfd99789611f4ab098a248d6366e52c9 (127.0.98.1:43257): tablet fa5b6883ed6d4de9a037942dd149bcba (table test-table [id=913aef4b5ea64c3b9522df5c58da12c0]) successfully deleted
I20260927 02:38:47.362937 2673 tablet_bootstrap.cc:492] T 95654a75a4954c9480568d5077d059bc P a62ac2b710b342938b3a8b0b911d9c1e: Bootstrap starting.
I20260927 02:38:47.363636 2673 tablet_bootstrap.cc:654] T 95654a75a4954c9480568d5077d059bc P a62ac2b710b342938b3a8b0b911d9c1e: Neither blocks nor log segments found. Creating new log.
I20260927 02:38:47.364240 2671 tablet_bootstrap.cc:492] T 95654a75a4954c9480568d5077d059bc P bfd99789611f4ab098a248d6366e52c9: Bootstrap starting.
I20260927 02:38:47.364885 2671 tablet_bootstrap.cc:654] T 95654a75a4954c9480568d5077d059bc P bfd99789611f4ab098a248d6366e52c9: Neither blocks nor log segments found. Creating new log.
I20260927 02:38:47.365639 2673 tablet_bootstrap.cc:492] T 95654a75a4954c9480568d5077d059bc P a62ac2b710b342938b3a8b0b911d9c1e: No bootstrap required, opened a new log
I20260927 02:38:47.365717 2673 ts_tablet_manager.cc:1397] T 95654a75a4954c9480568d5077d059bc P a62ac2b710b342938b3a8b0b911d9c1e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 02:38:47.365891 2671 tablet_bootstrap.cc:492] T 95654a75a4954c9480568d5077d059bc P bfd99789611f4ab098a248d6366e52c9: No bootstrap required, opened a new log
I20260927 02:38:47.365975 2671 ts_tablet_manager.cc:1397] T 95654a75a4954c9480568d5077d059bc P bfd99789611f4ab098a248d6366e52c9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:38:47.366128 2673 raft_consensus.cc:348] T 95654a75a4954c9480568d5077d059bc P a62ac2b710b342938b3a8b0b911d9c1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } }
I20260927 02:38:47.366222 2673 raft_consensus.cc:374] T 95654a75a4954c9480568d5077d059bc P a62ac2b710b342938b3a8b0b911d9c1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:38:47.366269 2673 raft_consensus.cc:729] T 95654a75a4954c9480568d5077d059bc P a62ac2b710b342938b3a8b0b911d9c1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a62ac2b710b342938b3a8b0b911d9c1e, State: Initialized, Role: FOLLOWER
I20260927 02:38:47.366411 2673 consensus_queue.cc:260] T 95654a75a4954c9480568d5077d059bc P a62ac2b710b342938b3a8b0b911d9c1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } }
I20260927 02:38:47.366369 2671 raft_consensus.cc:348] T 95654a75a4954c9480568d5077d059bc P bfd99789611f4ab098a248d6366e52c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } }
I20260927 02:38:47.366484 2671 raft_consensus.cc:374] T 95654a75a4954c9480568d5077d059bc P bfd99789611f4ab098a248d6366e52c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:38:47.366524 2671 raft_consensus.cc:729] T 95654a75a4954c9480568d5077d059bc P bfd99789611f4ab098a248d6366e52c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfd99789611f4ab098a248d6366e52c9, State: Initialized, Role: FOLLOWER
I20260927 02:38:47.366645 2671 consensus_queue.cc:260] T 95654a75a4954c9480568d5077d059bc P bfd99789611f4ab098a248d6366e52c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } }
I20260927 02:38:47.366889 2671 ts_tablet_manager.cc:1428] T 95654a75a4954c9480568d5077d059bc P bfd99789611f4ab098a248d6366e52c9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:38:47.367261 2673 ts_tablet_manager.cc:1428] T 95654a75a4954c9480568d5077d059bc P a62ac2b710b342938b3a8b0b911d9c1e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:38:47.431015 2632 raft_consensus.cc:482] T 95654a75a4954c9480568d5077d059bc P bfd99789611f4ab098a248d6366e52c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:38:47.431145 2632 raft_consensus.cc:504] T 95654a75a4954c9480568d5077d059bc P bfd99789611f4ab098a248d6366e52c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } }
I20260927 02:38:47.431643 2632 leader_election.cc:288] T 95654a75a4954c9480568d5077d059bc P bfd99789611f4ab098a248d6366e52c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f69698959a44a8ea710e230956c873e (127.0.98.2:45675), a62ac2b710b342938b3a8b0b911d9c1e (127.0.98.3:35547)
I20260927 02:38:47.432060 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95654a75a4954c9480568d5077d059bc" candidate_uuid: "bfd99789611f4ab098a248d6366e52c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" is_pre_election: true
I20260927 02:38:47.432085 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95654a75a4954c9480568d5077d059bc" candidate_uuid: "bfd99789611f4ab098a248d6366e52c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f69698959a44a8ea710e230956c873e" is_pre_election: true
I20260927 02:38:47.432230 2552 raft_consensus.cc:2454] T 95654a75a4954c9480568d5077d059bc P a62ac2b710b342938b3a8b0b911d9c1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfd99789611f4ab098a248d6366e52c9 in term 0.
I20260927 02:38:47.432240 2412 raft_consensus.cc:2454] T 95654a75a4954c9480568d5077d059bc P 7f69698959a44a8ea710e230956c873e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfd99789611f4ab098a248d6366e52c9 in term 0.
I20260927 02:38:47.432588 2203 leader_election.cc:302] T 95654a75a4954c9480568d5077d059bc P bfd99789611f4ab098a248d6366e52c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a62ac2b710b342938b3a8b0b911d9c1e, bfd99789611f4ab098a248d6366e52c9; no voters:
I20260927 02:38:47.432808 2632 raft_consensus.cc:2781] T 95654a75a4954c9480568d5077d059bc P bfd99789611f4ab098a248d6366e52c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:38:47.432870 2632 raft_consensus.cc:482] T 95654a75a4954c9480568d5077d059bc P bfd99789611f4ab098a248d6366e52c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:38:47.432906 2632 raft_consensus.cc:3037] T 95654a75a4954c9480568d5077d059bc P bfd99789611f4ab098a248d6366e52c9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:38:47.433851 2632 raft_consensus.cc:504] T 95654a75a4954c9480568d5077d059bc P bfd99789611f4ab098a248d6366e52c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } }
I20260927 02:38:47.434146 2632 leader_election.cc:288] T 95654a75a4954c9480568d5077d059bc P bfd99789611f4ab098a248d6366e52c9 [CANDIDATE]: Term 1 election: Requested vote from peers 7f69698959a44a8ea710e230956c873e (127.0.98.2:45675), a62ac2b710b342938b3a8b0b911d9c1e (127.0.98.3:35547)
I20260927 02:38:47.434460 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95654a75a4954c9480568d5077d059bc" candidate_uuid: "bfd99789611f4ab098a248d6366e52c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f69698959a44a8ea710e230956c873e"
I20260927 02:38:47.434559 2412 raft_consensus.cc:3037] T 95654a75a4954c9480568d5077d059bc P 7f69698959a44a8ea710e230956c873e [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:38:47.434525 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95654a75a4954c9480568d5077d059bc" candidate_uuid: "bfd99789611f4ab098a248d6366e52c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62ac2b710b342938b3a8b0b911d9c1e"
I20260927 02:38:47.434628 2552 raft_consensus.cc:3037] T 95654a75a4954c9480568d5077d059bc P a62ac2b710b342938b3a8b0b911d9c1e [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:38:47.435626 2412 raft_consensus.cc:2454] T 95654a75a4954c9480568d5077d059bc P 7f69698959a44a8ea710e230956c873e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfd99789611f4ab098a248d6366e52c9 in term 1.
I20260927 02:38:47.435655 2552 raft_consensus.cc:2454] T 95654a75a4954c9480568d5077d059bc P a62ac2b710b342938b3a8b0b911d9c1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfd99789611f4ab098a248d6366e52c9 in term 1.
I20260927 02:38:47.435914 2206 leader_election.cc:302] T 95654a75a4954c9480568d5077d059bc P bfd99789611f4ab098a248d6366e52c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f69698959a44a8ea710e230956c873e, bfd99789611f4ab098a248d6366e52c9; no voters:
I20260927 02:38:47.436182 2632 raft_consensus.cc:2781] T 95654a75a4954c9480568d5077d059bc P bfd99789611f4ab098a248d6366e52c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:38:47.436259 2632 raft_consensus.cc:686] T 95654a75a4954c9480568d5077d059bc P bfd99789611f4ab098a248d6366e52c9 [term 1 LEADER]: Becoming Leader. State: Replica: bfd99789611f4ab098a248d6366e52c9, State: Running, Role: LEADER
I20260927 02:38:47.436384 2632 consensus_queue.cc:237] T 95654a75a4954c9480568d5077d059bc P bfd99789611f4ab098a248d6366e52c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } }
I20260927 02:38:47.438167 2115 catalog_manager.cc:5495] T 95654a75a4954c9480568d5077d059bc P bfd99789611f4ab098a248d6366e52c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to bfd99789611f4ab098a248d6366e52c9 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "bfd99789611f4ab098a248d6366e52c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } health_report { overall_health: UNKNOWN } } }
I20260927 02:38:47.471766 2412 raft_consensus.cc:1261] T 95654a75a4954c9480568d5077d059bc P 7f69698959a44a8ea710e230956c873e [term 1 FOLLOWER]: Refusing update from remote peer bfd99789611f4ab098a248d6366e52c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:38:47.472064 2552 raft_consensus.cc:1261] T 95654a75a4954c9480568d5077d059bc P a62ac2b710b342938b3a8b0b911d9c1e [term 1 FOLLOWER]: Refusing update from remote peer bfd99789611f4ab098a248d6366e52c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:38:47.472177 2632 consensus_queue.cc:1035] T 95654a75a4954c9480568d5077d059bc P bfd99789611f4ab098a248d6366e52c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:38:47.477301 2658 consensus_queue.cc:1035] T 95654a75a4954c9480568d5077d059bc P bfd99789611f4ab098a248d6366e52c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 02:38:47.488371 2648 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 02:38:49.039407 2115 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:38:49.039548 2115 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:38:49.041827 2115 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 336ef4719fc24eb4aeb74a965882da30: Sending DeleteTablet for 3 replicas of tablet 95654a75a4954c9480568d5077d059bc
I20260927 02:38:49.042346 2252 tablet_service.cc:1514] Processing DeleteTablet for tablet 95654a75a4954c9480568d5077d059bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:38:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43668
I20260927 02:38:49.042515 2392 tablet_service.cc:1514] Processing DeleteTablet for tablet 95654a75a4954c9480568d5077d059bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:38:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41016
I20260927 02:38:49.042671 2532 tablet_service.cc:1514] Processing DeleteTablet for tablet 95654a75a4954c9480568d5077d059bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:38:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58810
I20260927 02:38:49.042948 2702 tablet_replica.cc:330] T 95654a75a4954c9480568d5077d059bc P 7f69698959a44a8ea710e230956c873e: stopping tablet replica
I20260927 02:38:49.043092 2702 raft_consensus.cc:2229] T 95654a75a4954c9480568d5077d059bc P 7f69698959a44a8ea710e230956c873e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:38:49.043226 2702 raft_consensus.cc:2258] T 95654a75a4954c9480568d5077d059bc P 7f69698959a44a8ea710e230956c873e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:38:49.043803 2702 ts_tablet_manager.cc:1905] T 95654a75a4954c9480568d5077d059bc P 7f69698959a44a8ea710e230956c873e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:38:49.044974 2701 tablet_replica.cc:330] T 95654a75a4954c9480568d5077d059bc P bfd99789611f4ab098a248d6366e52c9: stopping tablet replica
I20260927 02:38:49.045107 2701 raft_consensus.cc:2229] T 95654a75a4954c9480568d5077d059bc P bfd99789611f4ab098a248d6366e52c9 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:38:49.045306 2701 raft_consensus.cc:2258] T 95654a75a4954c9480568d5077d059bc P bfd99789611f4ab098a248d6366e52c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:38:49.046124 2701 ts_tablet_manager.cc:1905] T 95654a75a4954c9480568d5077d059bc P bfd99789611f4ab098a248d6366e52c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:38:49.046608 2702 ts_tablet_manager.cc:1918] T 95654a75a4954c9480568d5077d059bc P 7f69698959a44a8ea710e230956c873e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 02:38:49.046674 2702 log.cc:1196] T 95654a75a4954c9480568d5077d059bc P 7f69698959a44a8ea710e230956c873e: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-1/wal/wals/95654a75a4954c9480568d5077d059bc
I20260927 02:38:49.046917 2702 ts_tablet_manager.cc:1939] T 95654a75a4954c9480568d5077d059bc P 7f69698959a44a8ea710e230956c873e: Deleting consensus metadata
I20260927 02:38:49.047279 2703 tablet_replica.cc:330] T 95654a75a4954c9480568d5077d059bc P a62ac2b710b342938b3a8b0b911d9c1e: stopping tablet replica
I20260927 02:38:49.047410 2703 raft_consensus.cc:2229] T 95654a75a4954c9480568d5077d059bc P a62ac2b710b342938b3a8b0b911d9c1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:38:49.047523 2703 raft_consensus.cc:2258] T 95654a75a4954c9480568d5077d059bc P a62ac2b710b342938b3a8b0b911d9c1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:38:49.047654 2102 catalog_manager.cc:4841] TS 7f69698959a44a8ea710e230956c873e (127.0.98.2:45675): tablet 95654a75a4954c9480568d5077d059bc (table test-table [id=32602e0bab16478cbd024d3122415623]) successfully deleted
I20260927 02:38:49.048727 2701 ts_tablet_manager.cc:1918] T 95654a75a4954c9480568d5077d059bc P bfd99789611f4ab098a248d6366e52c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 02:38:49.048810 2701 log.cc:1196] T 95654a75a4954c9480568d5077d059bc P bfd99789611f4ab098a248d6366e52c9: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-0/wal/wals/95654a75a4954c9480568d5077d059bc
I20260927 02:38:49.049068 2701 ts_tablet_manager.cc:1939] T 95654a75a4954c9480568d5077d059bc P bfd99789611f4ab098a248d6366e52c9: Deleting consensus metadata
I20260927 02:38:49.049674 2099 catalog_manager.cc:4841] TS bfd99789611f4ab098a248d6366e52c9 (127.0.98.1:43257): tablet 95654a75a4954c9480568d5077d059bc (table test-table [id=32602e0bab16478cbd024d3122415623]) successfully deleted
I20260927 02:38:49.049979 2703 ts_tablet_manager.cc:1905] T 95654a75a4954c9480568d5077d059bc P a62ac2b710b342938b3a8b0b911d9c1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:38:49.052721 2703 ts_tablet_manager.cc:1918] T 95654a75a4954c9480568d5077d059bc P a62ac2b710b342938b3a8b0b911d9c1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 02:38:49.052791 2703 log.cc:1196] T 95654a75a4954c9480568d5077d059bc P a62ac2b710b342938b3a8b0b911d9c1e: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-2/wal/wals/95654a75a4954c9480568d5077d059bc
I20260927 02:38:49.053002 2703 ts_tablet_manager.cc:1939] T 95654a75a4954c9480568d5077d059bc P a62ac2b710b342938b3a8b0b911d9c1e: Deleting consensus metadata
I20260927 02:38:49.053581 2099 catalog_manager.cc:4841] TS a62ac2b710b342938b3a8b0b911d9c1e (127.0.98.3:35547): tablet 95654a75a4954c9480568d5077d059bc (table test-table [id=32602e0bab16478cbd024d3122415623]) successfully deleted
Sep 27 02:38:49 dist-test-slave-zv8z krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476729, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 02:38:49 dist-test-slave-zv8z krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476729, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:38:49.081453 2711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:49.071669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57192 (local address 127.0.98.62:33095)
0927 02:38:49.071995 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:49.072003 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:49.072017 (+ 14us) server_negotiation.cc:407] Connection header received
0927 02:38:49.072131 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:49.072135 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:49.072192 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:49.072284 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:49.073042 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:49.073896 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:49.075087 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:49.075420 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:49.075512 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:49.078166 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:49.078170 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:49.078172 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:49.078200 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:49.079917 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:49.080299 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:49.080303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:49.080305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:49.080394 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:49.080628 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:49.080632 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:49.080633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:49.080810 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:49.080926 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:49.081138 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 02:38:49.081255 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":96,"threads_started":1}
I20260927 02:38:49.084106 2115 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57192:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:38:49.084519 2115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:38:49.089360 2252 tablet_service.cc:1467] Processing CreateTablet for tablet 307dc9860ee34436b19c053766a55216 (DEFAULT_TABLE table=test-table [id=817d8f1f722b4422b8c8e9e938ac3eb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:38:49.089510 2532 tablet_service.cc:1467] Processing CreateTablet for tablet 307dc9860ee34436b19c053766a55216 (DEFAULT_TABLE table=test-table [id=817d8f1f722b4422b8c8e9e938ac3eb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:38:49.089536 2392 tablet_service.cc:1467] Processing CreateTablet for tablet 307dc9860ee34436b19c053766a55216 (DEFAULT_TABLE table=test-table [id=817d8f1f722b4422b8c8e9e938ac3eb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:38:49.089661 2252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 307dc9860ee34436b19c053766a55216. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:38:49.089762 2392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 307dc9860ee34436b19c053766a55216. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:38:49.089828 2532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 307dc9860ee34436b19c053766a55216. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:38:49.092125 2712 tablet_bootstrap.cc:492] T 307dc9860ee34436b19c053766a55216 P bfd99789611f4ab098a248d6366e52c9: Bootstrap starting.
I20260927 02:38:49.092496 2714 tablet_bootstrap.cc:492] T 307dc9860ee34436b19c053766a55216 P a62ac2b710b342938b3a8b0b911d9c1e: Bootstrap starting.
I20260927 02:38:49.093178 2712 tablet_bootstrap.cc:654] T 307dc9860ee34436b19c053766a55216 P bfd99789611f4ab098a248d6366e52c9: Neither blocks nor log segments found. Creating new log.
I20260927 02:38:49.093408 2713 tablet_bootstrap.cc:492] T 307dc9860ee34436b19c053766a55216 P 7f69698959a44a8ea710e230956c873e: Bootstrap starting.
I20260927 02:38:49.093595 2714 tablet_bootstrap.cc:654] T 307dc9860ee34436b19c053766a55216 P a62ac2b710b342938b3a8b0b911d9c1e: Neither blocks nor log segments found. Creating new log.
I20260927 02:38:49.094471 2713 tablet_bootstrap.cc:654] T 307dc9860ee34436b19c053766a55216 P 7f69698959a44a8ea710e230956c873e: Neither blocks nor log segments found. Creating new log.
I20260927 02:38:49.095446 2712 tablet_bootstrap.cc:492] T 307dc9860ee34436b19c053766a55216 P bfd99789611f4ab098a248d6366e52c9: No bootstrap required, opened a new log
I20260927 02:38:49.095446 2714 tablet_bootstrap.cc:492] T 307dc9860ee34436b19c053766a55216 P a62ac2b710b342938b3a8b0b911d9c1e: No bootstrap required, opened a new log
I20260927 02:38:49.095530 2714 ts_tablet_manager.cc:1397] T 307dc9860ee34436b19c053766a55216 P a62ac2b710b342938b3a8b0b911d9c1e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 02:38:49.095530 2712 ts_tablet_manager.cc:1397] T 307dc9860ee34436b19c053766a55216 P bfd99789611f4ab098a248d6366e52c9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 02:38:49.095938 2712 raft_consensus.cc:348] T 307dc9860ee34436b19c053766a55216 P bfd99789611f4ab098a248d6366e52c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } }
I20260927 02:38:49.095938 2714 raft_consensus.cc:348] T 307dc9860ee34436b19c053766a55216 P a62ac2b710b342938b3a8b0b911d9c1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } }
I20260927 02:38:49.096050 2712 raft_consensus.cc:374] T 307dc9860ee34436b19c053766a55216 P bfd99789611f4ab098a248d6366e52c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:38:49.096050 2714 raft_consensus.cc:374] T 307dc9860ee34436b19c053766a55216 P a62ac2b710b342938b3a8b0b911d9c1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:38:49.096088 2714 raft_consensus.cc:729] T 307dc9860ee34436b19c053766a55216 P a62ac2b710b342938b3a8b0b911d9c1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a62ac2b710b342938b3a8b0b911d9c1e, State: Initialized, Role: FOLLOWER
I20260927 02:38:49.096087 2712 raft_consensus.cc:729] T 307dc9860ee34436b19c053766a55216 P bfd99789611f4ab098a248d6366e52c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfd99789611f4ab098a248d6366e52c9, State: Initialized, Role: FOLLOWER
I20260927 02:38:49.096210 2712 consensus_queue.cc:260] T 307dc9860ee34436b19c053766a55216 P bfd99789611f4ab098a248d6366e52c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } }
I20260927 02:38:49.096221 2714 consensus_queue.cc:260] T 307dc9860ee34436b19c053766a55216 P a62ac2b710b342938b3a8b0b911d9c1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } }
I20260927 02:38:49.096624 2714 ts_tablet_manager.cc:1428] T 307dc9860ee34436b19c053766a55216 P a62ac2b710b342938b3a8b0b911d9c1e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:38:49.096655 2713 tablet_bootstrap.cc:492] T 307dc9860ee34436b19c053766a55216 P 7f69698959a44a8ea710e230956c873e: No bootstrap required, opened a new log
I20260927 02:38:49.097033 2712 ts_tablet_manager.cc:1428] T 307dc9860ee34436b19c053766a55216 P bfd99789611f4ab098a248d6366e52c9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:38:49.098430 2713 ts_tablet_manager.cc:1397] T 307dc9860ee34436b19c053766a55216 P 7f69698959a44a8ea710e230956c873e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260927 02:38:49.098869 2713 raft_consensus.cc:348] T 307dc9860ee34436b19c053766a55216 P 7f69698959a44a8ea710e230956c873e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } }
I20260927 02:38:49.099009 2713 raft_consensus.cc:374] T 307dc9860ee34436b19c053766a55216 P 7f69698959a44a8ea710e230956c873e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:38:49.099074 2713 raft_consensus.cc:729] T 307dc9860ee34436b19c053766a55216 P 7f69698959a44a8ea710e230956c873e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f69698959a44a8ea710e230956c873e, State: Initialized, Role: FOLLOWER
I20260927 02:38:49.099223 2713 consensus_queue.cc:260] T 307dc9860ee34436b19c053766a55216 P 7f69698959a44a8ea710e230956c873e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } }
I20260927 02:38:49.099627 2713 ts_tablet_manager.cc:1428] T 307dc9860ee34436b19c053766a55216 P 7f69698959a44a8ea710e230956c873e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:38:49.109341 2692 raft_consensus.cc:482] T 307dc9860ee34436b19c053766a55216 P bfd99789611f4ab098a248d6366e52c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:38:49.109445 2692 raft_consensus.cc:504] T 307dc9860ee34436b19c053766a55216 P bfd99789611f4ab098a248d6366e52c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } }
I20260927 02:38:49.109798 2692 leader_election.cc:288] T 307dc9860ee34436b19c053766a55216 P bfd99789611f4ab098a248d6366e52c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f69698959a44a8ea710e230956c873e (127.0.98.2:45675), a62ac2b710b342938b3a8b0b911d9c1e (127.0.98.3:35547)
I20260927 02:38:49.110172 2411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "307dc9860ee34436b19c053766a55216" candidate_uuid: "bfd99789611f4ab098a248d6366e52c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f69698959a44a8ea710e230956c873e" is_pre_election: true
I20260927 02:38:49.110186 2551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "307dc9860ee34436b19c053766a55216" candidate_uuid: "bfd99789611f4ab098a248d6366e52c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" is_pre_election: true
I20260927 02:38:49.110347 2411 raft_consensus.cc:2454] T 307dc9860ee34436b19c053766a55216 P 7f69698959a44a8ea710e230956c873e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfd99789611f4ab098a248d6366e52c9 in term 0.
I20260927 02:38:49.110364 2551 raft_consensus.cc:2454] T 307dc9860ee34436b19c053766a55216 P a62ac2b710b342938b3a8b0b911d9c1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfd99789611f4ab098a248d6366e52c9 in term 0.
I20260927 02:38:49.110634 2203 leader_election.cc:302] T 307dc9860ee34436b19c053766a55216 P bfd99789611f4ab098a248d6366e52c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a62ac2b710b342938b3a8b0b911d9c1e, bfd99789611f4ab098a248d6366e52c9; no voters:
I20260927 02:38:49.110780 2692 raft_consensus.cc:2781] T 307dc9860ee34436b19c053766a55216 P bfd99789611f4ab098a248d6366e52c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:38:49.110847 2692 raft_consensus.cc:482] T 307dc9860ee34436b19c053766a55216 P bfd99789611f4ab098a248d6366e52c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:38:49.110893 2692 raft_consensus.cc:3037] T 307dc9860ee34436b19c053766a55216 P bfd99789611f4ab098a248d6366e52c9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:38:49.111830 2692 raft_consensus.cc:504] T 307dc9860ee34436b19c053766a55216 P bfd99789611f4ab098a248d6366e52c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } }
I20260927 02:38:49.112129 2692 leader_election.cc:288] T 307dc9860ee34436b19c053766a55216 P bfd99789611f4ab098a248d6366e52c9 [CANDIDATE]: Term 1 election: Requested vote from peers 7f69698959a44a8ea710e230956c873e (127.0.98.2:45675), a62ac2b710b342938b3a8b0b911d9c1e (127.0.98.3:35547)
I20260927 02:38:49.112413 2411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "307dc9860ee34436b19c053766a55216" candidate_uuid: "bfd99789611f4ab098a248d6366e52c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f69698959a44a8ea710e230956c873e"
I20260927 02:38:49.112417 2551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "307dc9860ee34436b19c053766a55216" candidate_uuid: "bfd99789611f4ab098a248d6366e52c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62ac2b710b342938b3a8b0b911d9c1e"
I20260927 02:38:49.112536 2411 raft_consensus.cc:3037] T 307dc9860ee34436b19c053766a55216 P 7f69698959a44a8ea710e230956c873e [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:38:49.112538 2551 raft_consensus.cc:3037] T 307dc9860ee34436b19c053766a55216 P a62ac2b710b342938b3a8b0b911d9c1e [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:38:49.113384 2551 raft_consensus.cc:2454] T 307dc9860ee34436b19c053766a55216 P a62ac2b710b342938b3a8b0b911d9c1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfd99789611f4ab098a248d6366e52c9 in term 1.
I20260927 02:38:49.113446 2411 raft_consensus.cc:2454] T 307dc9860ee34436b19c053766a55216 P 7f69698959a44a8ea710e230956c873e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfd99789611f4ab098a248d6366e52c9 in term 1.
I20260927 02:38:49.113618 2203 leader_election.cc:302] T 307dc9860ee34436b19c053766a55216 P bfd99789611f4ab098a248d6366e52c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a62ac2b710b342938b3a8b0b911d9c1e, bfd99789611f4ab098a248d6366e52c9; no voters:
I20260927 02:38:49.113751 2692 raft_consensus.cc:2781] T 307dc9860ee34436b19c053766a55216 P bfd99789611f4ab098a248d6366e52c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:38:49.113838 2692 raft_consensus.cc:686] T 307dc9860ee34436b19c053766a55216 P bfd99789611f4ab098a248d6366e52c9 [term 1 LEADER]: Becoming Leader. State: Replica: bfd99789611f4ab098a248d6366e52c9, State: Running, Role: LEADER
I20260927 02:38:49.113981 2692 consensus_queue.cc:237] T 307dc9860ee34436b19c053766a55216 P bfd99789611f4ab098a248d6366e52c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } }
I20260927 02:38:49.115430 2114 catalog_manager.cc:5495] T 307dc9860ee34436b19c053766a55216 P bfd99789611f4ab098a248d6366e52c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to bfd99789611f4ab098a248d6366e52c9 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "bfd99789611f4ab098a248d6366e52c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } health_report { overall_health: UNKNOWN } } }
I20260927 02:38:49.127951 2722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:49.125245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43696 (local address 127.0.98.1:43257)
0927 02:38:49.125491 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:49.125498 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:49.125508 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:38:49.125542 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:49.125544 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:49.125579 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:49.125643 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:38:49.125840 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:49.126389 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:49.127147 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:49.127363 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:49.127431 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:49.127612 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:38:49.127674 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:38:49.127718 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:49.127777 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 02:38:49.127817 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":120,"threads_started":1}
I20260927 02:38:49.133029 2551 raft_consensus.cc:1261] T 307dc9860ee34436b19c053766a55216 P a62ac2b710b342938b3a8b0b911d9c1e [term 1 FOLLOWER]: Refusing update from remote peer bfd99789611f4ab098a248d6366e52c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:38:49.133029 2411 raft_consensus.cc:1261] T 307dc9860ee34436b19c053766a55216 P 7f69698959a44a8ea710e230956c873e [term 1 FOLLOWER]: Refusing update from remote peer bfd99789611f4ab098a248d6366e52c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:38:49.133371 2693 consensus_queue.cc:1035] T 307dc9860ee34436b19c053766a55216 P bfd99789611f4ab098a248d6366e52c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:38:49.133522 2683 consensus_queue.cc:1035] T 307dc9860ee34436b19c053766a55216 P bfd99789611f4ab098a248d6366e52c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:38:49.136633 2685 mvcc.cc:205] Tried to move back new op lower bound from 7333792682525179904 to 7333792682451820544. Current Snapshot: MvccSnapshot[applied={T|T < 7333792682525179904}]
I20260927 02:38:49.159772 2113 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:38:49.159929 2113 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:38:49.161729 2113 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 336ef4719fc24eb4aeb74a965882da30: Sending DeleteTablet for 3 replicas of tablet 307dc9860ee34436b19c053766a55216
I20260927 02:38:49.162160 2252 tablet_service.cc:1514] Processing DeleteTablet for tablet 307dc9860ee34436b19c053766a55216 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:38:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43668
I20260927 02:38:49.162217 2532 tablet_service.cc:1514] Processing DeleteTablet for tablet 307dc9860ee34436b19c053766a55216 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:38:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58810
I20260927 02:38:49.162216 2392 tablet_service.cc:1514] Processing DeleteTablet for tablet 307dc9860ee34436b19c053766a55216 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:38:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41016
I20260927 02:38:49.162379 2701 tablet_replica.cc:330] T 307dc9860ee34436b19c053766a55216 P bfd99789611f4ab098a248d6366e52c9: stopping tablet replica
I20260927 02:38:49.162426 2703 tablet_replica.cc:330] T 307dc9860ee34436b19c053766a55216 P a62ac2b710b342938b3a8b0b911d9c1e: stopping tablet replica
I20260927 02:38:49.162478 2701 raft_consensus.cc:2229] T 307dc9860ee34436b19c053766a55216 P bfd99789611f4ab098a248d6366e52c9 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:38:49.162521 2703 raft_consensus.cc:2229] T 307dc9860ee34436b19c053766a55216 P a62ac2b710b342938b3a8b0b911d9c1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:38:49.162600 2702 tablet_replica.cc:330] T 307dc9860ee34436b19c053766a55216 P 7f69698959a44a8ea710e230956c873e: stopping tablet replica
I20260927 02:38:49.162613 2703 raft_consensus.cc:2258] T 307dc9860ee34436b19c053766a55216 P a62ac2b710b342938b3a8b0b911d9c1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:38:49.162647 2701 raft_consensus.cc:2258] T 307dc9860ee34436b19c053766a55216 P bfd99789611f4ab098a248d6366e52c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:38:49.162695 2702 raft_consensus.cc:2229] T 307dc9860ee34436b19c053766a55216 P 7f69698959a44a8ea710e230956c873e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:38:49.162786 2702 raft_consensus.cc:2258] T 307dc9860ee34436b19c053766a55216 P 7f69698959a44a8ea710e230956c873e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:38:49.163005 2113 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57192:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 02:38:49.163353 2701 ts_tablet_manager.cc:1905] T 307dc9860ee34436b19c053766a55216 P bfd99789611f4ab098a248d6366e52c9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 02:38:49.163383 2113 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:38:49.163408 2703 ts_tablet_manager.cc:1905] T 307dc9860ee34436b19c053766a55216 P a62ac2b710b342938b3a8b0b911d9c1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:38:49.163472 2702 ts_tablet_manager.cc:1905] T 307dc9860ee34436b19c053766a55216 P 7f69698959a44a8ea710e230956c873e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:38:49.165370 2703 ts_tablet_manager.cc:1918] T 307dc9860ee34436b19c053766a55216 P a62ac2b710b342938b3a8b0b911d9c1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:38:49.165419 2703 log.cc:1196] T 307dc9860ee34436b19c053766a55216 P a62ac2b710b342938b3a8b0b911d9c1e: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-2/wal/wals/307dc9860ee34436b19c053766a55216
I20260927 02:38:49.165582 2703 ts_tablet_manager.cc:1939] T 307dc9860ee34436b19c053766a55216 P a62ac2b710b342938b3a8b0b911d9c1e: Deleting consensus metadata
I20260927 02:38:49.165634 2701 ts_tablet_manager.cc:1918] T 307dc9860ee34436b19c053766a55216 P bfd99789611f4ab098a248d6366e52c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:38:49.165634 2702 ts_tablet_manager.cc:1918] T 307dc9860ee34436b19c053766a55216 P 7f69698959a44a8ea710e230956c873e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:38:49.165696 2702 log.cc:1196] T 307dc9860ee34436b19c053766a55216 P 7f69698959a44a8ea710e230956c873e: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-1/wal/wals/307dc9860ee34436b19c053766a55216
I20260927 02:38:49.165696 2701 log.cc:1196] T 307dc9860ee34436b19c053766a55216 P bfd99789611f4ab098a248d6366e52c9: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-0/wal/wals/307dc9860ee34436b19c053766a55216
I20260927 02:38:49.165920 2701 ts_tablet_manager.cc:1939] T 307dc9860ee34436b19c053766a55216 P bfd99789611f4ab098a248d6366e52c9: Deleting consensus metadata
I20260927 02:38:49.165938 2702 ts_tablet_manager.cc:1939] T 307dc9860ee34436b19c053766a55216 P 7f69698959a44a8ea710e230956c873e: Deleting consensus metadata
I20260927 02:38:49.166200 2099 catalog_manager.cc:4841] TS a62ac2b710b342938b3a8b0b911d9c1e (127.0.98.3:35547): tablet 307dc9860ee34436b19c053766a55216 (table test-table [id=817d8f1f722b4422b8c8e9e938ac3eb4]) successfully deleted
I20260927 02:38:49.166541 2102 catalog_manager.cc:4841] TS 7f69698959a44a8ea710e230956c873e (127.0.98.2:45675): tablet 307dc9860ee34436b19c053766a55216 (table test-table [id=817d8f1f722b4422b8c8e9e938ac3eb4]) successfully deleted
I20260927 02:38:49.167061 2099 catalog_manager.cc:4841] TS bfd99789611f4ab098a248d6366e52c9 (127.0.98.1:43257): tablet 307dc9860ee34436b19c053766a55216 (table test-table [id=817d8f1f722b4422b8c8e9e938ac3eb4]) successfully deleted
I20260927 02:38:49.169559 2252 tablet_service.cc:1467] Processing CreateTablet for tablet f4e278295396432d8ba5c0cd4bb3e758 (DEFAULT_TABLE table=test-table [id=ec5d280f7a5d43d58be60efe224ecdb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:38:49.169625 2392 tablet_service.cc:1467] Processing CreateTablet for tablet f4e278295396432d8ba5c0cd4bb3e758 (DEFAULT_TABLE table=test-table [id=ec5d280f7a5d43d58be60efe224ecdb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:38:49.169718 2532 tablet_service.cc:1467] Processing CreateTablet for tablet f4e278295396432d8ba5c0cd4bb3e758 (DEFAULT_TABLE table=test-table [id=ec5d280f7a5d43d58be60efe224ecdb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:38:49.169818 2252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4e278295396432d8ba5c0cd4bb3e758. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:38:49.169838 2392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4e278295396432d8ba5c0cd4bb3e758. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:38:49.169941 2532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4e278295396432d8ba5c0cd4bb3e758. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:38:49.171602 2712 tablet_bootstrap.cc:492] T f4e278295396432d8ba5c0cd4bb3e758 P bfd99789611f4ab098a248d6366e52c9: Bootstrap starting.
I20260927 02:38:49.172006 2714 tablet_bootstrap.cc:492] T f4e278295396432d8ba5c0cd4bb3e758 P a62ac2b710b342938b3a8b0b911d9c1e: Bootstrap starting.
I20260927 02:38:49.172119 2713 tablet_bootstrap.cc:492] T f4e278295396432d8ba5c0cd4bb3e758 P 7f69698959a44a8ea710e230956c873e: Bootstrap starting.
I20260927 02:38:49.172399 2712 tablet_bootstrap.cc:654] T f4e278295396432d8ba5c0cd4bb3e758 P bfd99789611f4ab098a248d6366e52c9: Neither blocks nor log segments found. Creating new log.
I20260927 02:38:49.172811 2714 tablet_bootstrap.cc:654] T f4e278295396432d8ba5c0cd4bb3e758 P a62ac2b710b342938b3a8b0b911d9c1e: Neither blocks nor log segments found. Creating new log.
I20260927 02:38:49.172900 2713 tablet_bootstrap.cc:654] T f4e278295396432d8ba5c0cd4bb3e758 P 7f69698959a44a8ea710e230956c873e: Neither blocks nor log segments found. Creating new log.
I20260927 02:38:49.173245 2712 tablet_bootstrap.cc:492] T f4e278295396432d8ba5c0cd4bb3e758 P bfd99789611f4ab098a248d6366e52c9: No bootstrap required, opened a new log
I20260927 02:38:49.173323 2712 ts_tablet_manager.cc:1397] T f4e278295396432d8ba5c0cd4bb3e758 P bfd99789611f4ab098a248d6366e52c9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:38:49.173631 2712 raft_consensus.cc:348] T f4e278295396432d8ba5c0cd4bb3e758 P bfd99789611f4ab098a248d6366e52c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } }
I20260927 02:38:49.173736 2712 raft_consensus.cc:374] T f4e278295396432d8ba5c0cd4bb3e758 P bfd99789611f4ab098a248d6366e52c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:38:49.173766 2712 raft_consensus.cc:729] T f4e278295396432d8ba5c0cd4bb3e758 P bfd99789611f4ab098a248d6366e52c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfd99789611f4ab098a248d6366e52c9, State: Initialized, Role: FOLLOWER
I20260927 02:38:49.173866 2713 tablet_bootstrap.cc:492] T f4e278295396432d8ba5c0cd4bb3e758 P 7f69698959a44a8ea710e230956c873e: No bootstrap required, opened a new log
I20260927 02:38:49.173933 2713 ts_tablet_manager.cc:1397] T f4e278295396432d8ba5c0cd4bb3e758 P 7f69698959a44a8ea710e230956c873e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:38:49.173892 2712 consensus_queue.cc:260] T f4e278295396432d8ba5c0cd4bb3e758 P bfd99789611f4ab098a248d6366e52c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } }
I20260927 02:38:49.174023 2714 tablet_bootstrap.cc:492] T f4e278295396432d8ba5c0cd4bb3e758 P a62ac2b710b342938b3a8b0b911d9c1e: No bootstrap required, opened a new log
I20260927 02:38:49.174089 2714 ts_tablet_manager.cc:1397] T f4e278295396432d8ba5c0cd4bb3e758 P a62ac2b710b342938b3a8b0b911d9c1e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:38:49.174147 2712 ts_tablet_manager.cc:1428] T f4e278295396432d8ba5c0cd4bb3e758 P bfd99789611f4ab098a248d6366e52c9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 02:38:49.174316 2713 raft_consensus.cc:348] T f4e278295396432d8ba5c0cd4bb3e758 P 7f69698959a44a8ea710e230956c873e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } }
I20260927 02:38:49.174433 2713 raft_consensus.cc:374] T f4e278295396432d8ba5c0cd4bb3e758 P 7f69698959a44a8ea710e230956c873e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:38:49.174464 2713 raft_consensus.cc:729] T f4e278295396432d8ba5c0cd4bb3e758 P 7f69698959a44a8ea710e230956c873e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f69698959a44a8ea710e230956c873e, State: Initialized, Role: FOLLOWER
I20260927 02:38:49.174506 2714 raft_consensus.cc:348] T f4e278295396432d8ba5c0cd4bb3e758 P a62ac2b710b342938b3a8b0b911d9c1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } }
I20260927 02:38:49.174607 2714 raft_consensus.cc:374] T f4e278295396432d8ba5c0cd4bb3e758 P a62ac2b710b342938b3a8b0b911d9c1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:38:49.174568 2713 consensus_queue.cc:260] T f4e278295396432d8ba5c0cd4bb3e758 P 7f69698959a44a8ea710e230956c873e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } }
I20260927 02:38:49.174639 2714 raft_consensus.cc:729] T f4e278295396432d8ba5c0cd4bb3e758 P a62ac2b710b342938b3a8b0b911d9c1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a62ac2b710b342938b3a8b0b911d9c1e, State: Initialized, Role: FOLLOWER
I20260927 02:38:49.174742 2714 consensus_queue.cc:260] T f4e278295396432d8ba5c0cd4bb3e758 P a62ac2b710b342938b3a8b0b911d9c1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } }
I20260927 02:38:49.174800 2713 ts_tablet_manager.cc:1428] T f4e278295396432d8ba5c0cd4bb3e758 P 7f69698959a44a8ea710e230956c873e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:38:49.174979 2714 ts_tablet_manager.cc:1428] T f4e278295396432d8ba5c0cd4bb3e758 P a62ac2b710b342938b3a8b0b911d9c1e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 02:38:49.538060 2718 raft_consensus.cc:482] T f4e278295396432d8ba5c0cd4bb3e758 P a62ac2b710b342938b3a8b0b911d9c1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:38:49.538198 2718 raft_consensus.cc:504] T f4e278295396432d8ba5c0cd4bb3e758 P a62ac2b710b342938b3a8b0b911d9c1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } }
I20260927 02:38:49.539270 2718 leader_election.cc:288] T f4e278295396432d8ba5c0cd4bb3e758 P a62ac2b710b342938b3a8b0b911d9c1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bfd99789611f4ab098a248d6366e52c9 (127.0.98.1:43257), 7f69698959a44a8ea710e230956c873e (127.0.98.2:45675)
I20260927 02:38:49.544741 2722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:49.539626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35117 (local address 127.0.98.1:43257)
0927 02:38:49.539772 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:49.539776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:49.540055 (+ 279us) server_negotiation.cc:407] Connection header received
0927 02:38:49.540132 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:49.540135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:49.540185 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:49.540274 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:38:49.540746 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:49.541571 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:49.543320 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:49.543984 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:49.544088 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:49.544160 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:49.544535 (+ 375us) server_negotiation.cc:299] Negotiation successful
0927 02:38:49.544608 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 02:38:49.544752 2731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:49.539626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52057 (local address 127.0.98.2:45675)
0927 02:38:49.539925 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:49.539937 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:49.539971 (+ 34us) server_negotiation.cc:407] Connection header received
0927 02:38:49.540132 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:49.540135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:49.540185 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:49.540291 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:38:49.540732 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:49.541575 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:49.543313 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:49.543977 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:49.544088 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:49.544164 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:49.544534 (+ 370us) server_negotiation.cc:299] Negotiation successful
0927 02:38:49.544608 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":78,"threads_started":1}
I20260927 02:38:49.544813 2729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:49.539537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45675 (local address 127.0.98.3:52057)
0927 02:38:49.539892 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:49.539907 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:49.540026 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:49.540327 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:49.540330 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:49.540378 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:38:49.540567 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:49.540574 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:49.541728 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:49.541732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:49.543174 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:49.543187 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:49.544151 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:49.544154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:49.544367 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:49.544439 (+ 72us) client_negotiation.cc:769] Sending connection context
0927 02:38:49.544518 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 02:38:49.544622 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":145,"spinlock_wait_cycles":26496,"thread_start_us":107,"threads_started":1}
I20260927 02:38:49.545199 2730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:49.539537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43257 (local address 127.0.98.3:35117)
0927 02:38:49.539971 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:49.539982 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:49.540055 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:49.540327 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:49.540330 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:49.540376 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:38:49.540569 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:49.540574 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:49.541769 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:49.541772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:49.543174 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:49.543187 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:49.544128 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:49.544132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:49.544365 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:49.544437 (+ 72us) client_negotiation.cc:769] Sending connection context
0927 02:38:49.544518 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 02:38:49.544622 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":181,"spinlock_wait_cycles":31360,"thread_start_us":99,"threads_started":1}
I20260927 02:38:49.545310 2411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4e278295396432d8ba5c0cd4bb3e758" candidate_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f69698959a44a8ea710e230956c873e" is_pre_election: true
I20260927 02:38:49.545463 2411 raft_consensus.cc:2454] T f4e278295396432d8ba5c0cd4bb3e758 P 7f69698959a44a8ea710e230956c873e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a62ac2b710b342938b3a8b0b911d9c1e in term 0.
I20260927 02:38:49.545784 2272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4e278295396432d8ba5c0cd4bb3e758" candidate_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfd99789611f4ab098a248d6366e52c9" is_pre_election: true
I20260927 02:38:49.546007 2272 raft_consensus.cc:2454] T f4e278295396432d8ba5c0cd4bb3e758 P bfd99789611f4ab098a248d6366e52c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a62ac2b710b342938b3a8b0b911d9c1e in term 0.
I20260927 02:38:49.546160 2486 leader_election.cc:302] T f4e278295396432d8ba5c0cd4bb3e758 P a62ac2b710b342938b3a8b0b911d9c1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f69698959a44a8ea710e230956c873e, a62ac2b710b342938b3a8b0b911d9c1e; no voters:
I20260927 02:38:49.546538 2718 raft_consensus.cc:2781] T f4e278295396432d8ba5c0cd4bb3e758 P a62ac2b710b342938b3a8b0b911d9c1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:38:49.546602 2718 raft_consensus.cc:482] T f4e278295396432d8ba5c0cd4bb3e758 P a62ac2b710b342938b3a8b0b911d9c1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:38:49.546643 2718 raft_consensus.cc:3037] T f4e278295396432d8ba5c0cd4bb3e758 P a62ac2b710b342938b3a8b0b911d9c1e [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:38:49.547598 2718 raft_consensus.cc:504] T f4e278295396432d8ba5c0cd4bb3e758 P a62ac2b710b342938b3a8b0b911d9c1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } }
I20260927 02:38:49.547891 2718 leader_election.cc:288] T f4e278295396432d8ba5c0cd4bb3e758 P a62ac2b710b342938b3a8b0b911d9c1e [CANDIDATE]: Term 1 election: Requested vote from peers bfd99789611f4ab098a248d6366e52c9 (127.0.98.1:43257), 7f69698959a44a8ea710e230956c873e (127.0.98.2:45675)
I20260927 02:38:49.548194 2272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4e278295396432d8ba5c0cd4bb3e758" candidate_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfd99789611f4ab098a248d6366e52c9"
I20260927 02:38:49.548309 2272 raft_consensus.cc:3037] T f4e278295396432d8ba5c0cd4bb3e758 P bfd99789611f4ab098a248d6366e52c9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:38:49.548273 2411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4e278295396432d8ba5c0cd4bb3e758" candidate_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f69698959a44a8ea710e230956c873e"
I20260927 02:38:49.548395 2411 raft_consensus.cc:3037] T f4e278295396432d8ba5c0cd4bb3e758 P 7f69698959a44a8ea710e230956c873e [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:38:49.549207 2272 raft_consensus.cc:2454] T f4e278295396432d8ba5c0cd4bb3e758 P bfd99789611f4ab098a248d6366e52c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a62ac2b710b342938b3a8b0b911d9c1e in term 1.
I20260927 02:38:49.549250 2411 raft_consensus.cc:2454] T f4e278295396432d8ba5c0cd4bb3e758 P 7f69698959a44a8ea710e230956c873e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a62ac2b710b342938b3a8b0b911d9c1e in term 1.
I20260927 02:38:49.549492 2483 leader_election.cc:302] T f4e278295396432d8ba5c0cd4bb3e758 P a62ac2b710b342938b3a8b0b911d9c1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a62ac2b710b342938b3a8b0b911d9c1e, bfd99789611f4ab098a248d6366e52c9; no voters:
I20260927 02:38:49.549655 2718 raft_consensus.cc:2781] T f4e278295396432d8ba5c0cd4bb3e758 P a62ac2b710b342938b3a8b0b911d9c1e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:38:49.549878 2718 raft_consensus.cc:686] T f4e278295396432d8ba5c0cd4bb3e758 P a62ac2b710b342938b3a8b0b911d9c1e [term 1 LEADER]: Becoming Leader. State: Replica: a62ac2b710b342938b3a8b0b911d9c1e, State: Running, Role: LEADER
I20260927 02:38:49.550107 2718 consensus_queue.cc:237] T f4e278295396432d8ba5c0cd4bb3e758 P a62ac2b710b342938b3a8b0b911d9c1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } }
I20260927 02:38:49.552206 2113 catalog_manager.cc:5495] T f4e278295396432d8ba5c0cd4bb3e758 P a62ac2b710b342938b3a8b0b911d9c1e reported cstate change: term changed from 0 to 1, leader changed from <none> to a62ac2b710b342938b3a8b0b911d9c1e (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a62ac2b710b342938b3a8b0b911d9c1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35547 } health_report { overall_health: HEALTHY } } }
I20260927 02:38:49.631735 2735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:49.628100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58814 (local address 127.0.98.3:35547)
0927 02:38:49.628412 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:49.628418 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:49.628430 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:38:49.628494 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:49.628496 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:49.628544 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:49.628647 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:38:49.628899 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:49.629404 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:49.630279 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:49.630521 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:49.630597 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:49.630926 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:38:49.631355 (+ 429us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:38:49.631437 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:49.631506 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 02:38:49.631561 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":154,"threads_started":1}
I20260927 02:38:49.640771 2722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:49.638068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34071 (local address 127.0.98.1:43257)
0927 02:38:49.638198 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:49.638203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:49.638369 (+ 166us) server_negotiation.cc:407] Connection header received
0927 02:38:49.638442 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:49.638444 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:49.638490 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:49.638548 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:38:49.638731 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:49.639240 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:49.639991 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:49.640196 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:49.640269 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:49.640463 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:38:49.640522 (+ 59us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:38:49.640568 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:49.640626 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 02:38:49.640667 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 02:38:49.640781 2738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:49.637982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43257 (local address 127.0.98.3:34071)
0927 02:38:49.638237 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:49.638277 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:49.638366 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:49.638560 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:49.638562 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:49.638568 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:38:49.638653 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:49.638657 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:49.639320 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:49.639322 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:49.639896 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:49.639903 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:49.640272 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:49.640274 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:49.640338 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:49.640373 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:38:49.640575 (+ 202us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:38:49.640579 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:38:49.640619 (+ 40us) client_negotiation.cc:240] Negotiation successful
0927 02:38:49.640673 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":96,"threads_started":1}
I20260927 02:38:49.646224 2411 raft_consensus.cc:1261] T f4e278295396432d8ba5c0cd4bb3e758 P 7f69698959a44a8ea710e230956c873e [term 1 FOLLOWER]: Refusing update from remote peer a62ac2b710b342938b3a8b0b911d9c1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:38:49.646317 2272 raft_consensus.cc:1261] T f4e278295396432d8ba5c0cd4bb3e758 P bfd99789611f4ab098a248d6366e52c9 [term 1 FOLLOWER]: Refusing update from remote peer a62ac2b710b342938b3a8b0b911d9c1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:38:49.646746 2732 consensus_queue.cc:1035] T f4e278295396432d8ba5c0cd4bb3e758 P a62ac2b710b342938b3a8b0b911d9c1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f69698959a44a8ea710e230956c873e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:38:49.646915 2718 consensus_queue.cc:1035] T f4e278295396432d8ba5c0cd4bb3e758 P a62ac2b710b342938b3a8b0b911d9c1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfd99789611f4ab098a248d6366e52c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:38:49.664776 2735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:49.660739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42267 (local address 127.0.98.3:35547)
0927 02:38:49.661815 (+ 1076us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:49.661819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:49.661842 (+ 23us) server_negotiation.cc:407] Connection header received
0927 02:38:49.661880 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:49.661883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:49.661928 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:49.662001 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:38:49.662300 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:49.663010 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:49.663888 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:49.664106 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:49.664189 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:49.664402 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:38:49.664468 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:38:49.664523 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:49.664605 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 02:38:49.664650 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1000}
I20260927 02:38:49.664793 2744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:49.660586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35547 (local address 127.0.98.1:42267)
0927 02:38:49.660852 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:49.660867 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:49.660953 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:49.662034 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:49.662037 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:49.662048 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:38:49.662164 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:49.662171 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:49.663092 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:49.663096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:49.663785 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:49.663794 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:49.664196 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:49.664199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:49.664279 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:49.664303 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:38:49.664535 (+ 232us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:38:49.664541 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 02:38:49.664596 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 02:38:49.664672 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":50,"threads_started":1}
W20260927 02:38:49.665380 2684 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 02:38:51.219265 2113 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:38:51.219419 2113 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:38:51.221598 2113 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 336ef4719fc24eb4aeb74a965882da30: Sending DeleteTablet for 3 replicas of tablet f4e278295396432d8ba5c0cd4bb3e758
I20260927 02:38:51.222107 2392 tablet_service.cc:1514] Processing DeleteTablet for tablet f4e278295396432d8ba5c0cd4bb3e758 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:38:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41016
I20260927 02:38:51.222106 2252 tablet_service.cc:1514] Processing DeleteTablet for tablet f4e278295396432d8ba5c0cd4bb3e758 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:38:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43668
I20260927 02:38:51.222129 2532 tablet_service.cc:1514] Processing DeleteTablet for tablet f4e278295396432d8ba5c0cd4bb3e758 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:38:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58810
I20260927 02:38:51.222460 2772 tablet_replica.cc:330] T f4e278295396432d8ba5c0cd4bb3e758 P bfd99789611f4ab098a248d6366e52c9: stopping tablet replica
I20260927 02:38:51.222491 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 2187
I20260927 02:38:51.223102 2771 tablet_replica.cc:330] T f4e278295396432d8ba5c0cd4bb3e758 P 7f69698959a44a8ea710e230956c873e: stopping tablet replica
I20260927 02:38:51.223229 2771 raft_consensus.cc:2229] T f4e278295396432d8ba5c0cd4bb3e758 P 7f69698959a44a8ea710e230956c873e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:38:51.223341 2771 raft_consensus.cc:2258] T f4e278295396432d8ba5c0cd4bb3e758 P 7f69698959a44a8ea710e230956c873e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:38:51.222523 2770 tablet_replica.cc:330] T f4e278295396432d8ba5c0cd4bb3e758 P a62ac2b710b342938b3a8b0b911d9c1e: stopping tablet replica
I20260927 02:38:51.223526 2770 raft_consensus.cc:2229] T f4e278295396432d8ba5c0cd4bb3e758 P a62ac2b710b342938b3a8b0b911d9c1e [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:38:51.223838 2770 raft_consensus.cc:2258] T f4e278295396432d8ba5c0cd4bb3e758 P a62ac2b710b342938b3a8b0b911d9c1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:38:51.224107 2771 ts_tablet_manager.cc:1905] T f4e278295396432d8ba5c0cd4bb3e758 P 7f69698959a44a8ea710e230956c873e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:38:51.224418 2770 ts_tablet_manager.cc:1905] T f4e278295396432d8ba5c0cd4bb3e758 P a62ac2b710b342938b3a8b0b911d9c1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:38:51.226878 2771 ts_tablet_manager.cc:1918] T f4e278295396432d8ba5c0cd4bb3e758 P 7f69698959a44a8ea710e230956c873e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 02:38:51.226877 2770 ts_tablet_manager.cc:1918] T f4e278295396432d8ba5c0cd4bb3e758 P a62ac2b710b342938b3a8b0b911d9c1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 02:38:51.226961 2770 log.cc:1196] T f4e278295396432d8ba5c0cd4bb3e758 P a62ac2b710b342938b3a8b0b911d9c1e: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-2/wal/wals/f4e278295396432d8ba5c0cd4bb3e758
I20260927 02:38:51.227025 2771 log.cc:1196] T f4e278295396432d8ba5c0cd4bb3e758 P 7f69698959a44a8ea710e230956c873e: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790476718023131-392-0/minicluster-data/ts-1/wal/wals/f4e278295396432d8ba5c0cd4bb3e758
I20260927 02:38:51.227229 2770 ts_tablet_manager.cc:1939] T f4e278295396432d8ba5c0cd4bb3e758 P a62ac2b710b342938b3a8b0b911d9c1e: Deleting consensus metadata
I20260927 02:38:51.227259 2771 ts_tablet_manager.cc:1939] T f4e278295396432d8ba5c0cd4bb3e758 P 7f69698959a44a8ea710e230956c873e: Deleting consensus metadata
I20260927 02:38:51.227944 2102 catalog_manager.cc:4841] TS 7f69698959a44a8ea710e230956c873e (127.0.98.2:45675): tablet f4e278295396432d8ba5c0cd4bb3e758 (table test-table [id=ec5d280f7a5d43d58be60efe224ecdb8]) successfully deleted
I20260927 02:38:51.227989 2099 catalog_manager.cc:4841] TS a62ac2b710b342938b3a8b0b911d9c1e (127.0.98.3:35547): tablet f4e278295396432d8ba5c0cd4bb3e758 (table test-table [id=ec5d280f7a5d43d58be60efe224ecdb8]) successfully deleted
W20260927 02:38:51.229437 2099 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43257 (error 108)
I20260927 02:38:51.229614 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 2327
I20260927 02:38:51.231626 2774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:51.231250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43257 (local address 127.0.0.1:40616)
0927 02:38:51.231473 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:51.231538 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":97,"threads_started":1}
W20260927 02:38:51.231745 2099 catalog_manager.cc:4584] TS bfd99789611f4ab098a248d6366e52c9 (127.0.98.1:43257): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f4e278295396432d8ba5c0cd4bb3e758: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43257: connect: Connection refused (error 111)
I20260927 02:38:51.234699 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 2467
I20260927 02:38:51.240626 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 2083
2026-09-27T02:38:51Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6552 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:38:51 dist-test-slave-zv8z krb5kdc[2777](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:38:51 dist-test-slave-zv8z krb5kdc[2777](info): set up 2 sockets
Sep 27 02:38:51 dist-test-slave-zv8z krb5kdc[2777](info): commencing operation
krb5kdc: starting...
W20260927 02:38:52.202018 392 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/krb5kdc/test-user.keytab.
Sep 27 02:38:52 dist-test-slave-zv8z krb5kdc[2777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:38:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:38:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:38:52.316749 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43123
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43123
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:38:52.424422 2790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:38:52.424679 2790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:38:52.424718 2790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:38:52.428716 2790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:38:52.428781 2790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:38:52.428798 2790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:38:52.428813 2790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:38:52.428826 2790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:38:52.434113 2790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:38:52.435161 2790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:38:52.436168 2790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:38:52.444674 2790 server_base.cc:1034] running on GCE node
W20260927 02:38:52.444782 2798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:38:52.444782 2795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:38:52.444952 2796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:38:52.445521 2790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:38:52.446611 2790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:38:52.447773 2790 hybrid_clock.cc:648] HybridClock initialized: now 1790476732447757 us; error 29 us; skew 500 ppm
Sep 27 02:38:52 dist-test-slave-zv8z krb5kdc[2777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:38:52.450402 2790 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:38:52.451231 2790 webserver.cc:460] Webserver started at http://127.0.98.62:35417/ using document root <none> and password file <none>
I20260927 02:38:52.451761 2790 fs_manager.cc:359] Metadata directory not provided
I20260927 02:38:52.451835 2790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:38:52.452023 2790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:38:52.453944 2790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/master-0/data/instance:
uuid: "403970ae0c2e4e2da25df5a04b7b7e7c"
format_stamp: "Formatted at 2026-09-27 02:38:52 on dist-test-slave-zv8z"
server_key: "998e5e54ab96f4e545b25eda17ba7204"
server_key_iv: "ab5cf8fca20313da65953c4495587c17"
server_key_version: "encryptionkey@0"
I20260927 02:38:52.454483 2790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/master-0/wal/instance:
uuid: "403970ae0c2e4e2da25df5a04b7b7e7c"
format_stamp: "Formatted at 2026-09-27 02:38:52 on dist-test-slave-zv8z"
server_key: "998e5e54ab96f4e545b25eda17ba7204"
server_key_iv: "ab5cf8fca20313da65953c4495587c17"
server_key_version: "encryptionkey@0"
I20260927 02:38:52.457906 2790 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 02:38:52.460165 2805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:52.461311 2790 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 02:38:52.461417 2790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "403970ae0c2e4e2da25df5a04b7b7e7c"
format_stamp: "Formatted at 2026-09-27 02:38:52 on dist-test-slave-zv8z"
server_key: "998e5e54ab96f4e545b25eda17ba7204"
server_key_iv: "ab5cf8fca20313da65953c4495587c17"
server_key_version: "encryptionkey@0"
I20260927 02:38:52.461500 2790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:38:52.470877 2790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:38:52.473721 2790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:38:52.473877 2790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:38:52.480187 2790 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43123
I20260927 02:38:52.480237 2857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43123 every 8 connection(s)
I20260927 02:38:52.481063 2790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:38:52.482386 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 2790
I20260927 02:38:52.482489 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/master-0/wal/instance
I20260927 02:38:52.482748 392 external_mini_cluster.cc:1428] Setting key b3a4747e81bcdecf6f9874f03d90582e
I20260927 02:38:52.483845 2858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:38:52.489660 2858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 403970ae0c2e4e2da25df5a04b7b7e7c: Bootstrap starting.
Sep 27 02:38:52 dist-test-slave-zv8z krb5kdc[2777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:38:52.492063 2858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 403970ae0c2e4e2da25df5a04b7b7e7c: Neither blocks nor log segments found. Creating new log.
I20260927 02:38:52.492718 2858 log.cc:824] T 00000000000000000000000000000000 P 403970ae0c2e4e2da25df5a04b7b7e7c: Log is configured to *not* fsync() on all Append() calls
I20260927 02:38:52.494769 2858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 403970ae0c2e4e2da25df5a04b7b7e7c: No bootstrap required, opened a new log
I20260927 02:38:52.496699 2861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:52.484166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34380 (local address 127.0.98.62:43123)
0927 02:38:52.484604 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:52.484620 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:52.484653 (+ 33us) server_negotiation.cc:407] Connection header received
0927 02:38:52.485258 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:52.485273 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:52.485592 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:52.485856 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:52.486783 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:52.487738 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:52.488860 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:52.489311 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:52.489423 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:52.491989 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:52.491994 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:52.492006 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:52.492031 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:52.493669 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:52.494096 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:52.494110 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:52.494115 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:52.494245 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:52.494547 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:52.494550 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:52.494552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:52.495022 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:52.495214 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:52.495501 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 02:38:52.495785 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":140,"threads_started":1}
I20260927 02:38:52.498907 2858 raft_consensus.cc:348] T 00000000000000000000000000000000 P 403970ae0c2e4e2da25df5a04b7b7e7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "403970ae0c2e4e2da25df5a04b7b7e7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43123 } }
I20260927 02:38:52.499120 2858 raft_consensus.cc:374] T 00000000000000000000000000000000 P 403970ae0c2e4e2da25df5a04b7b7e7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:38:52.499167 2858 raft_consensus.cc:729] T 00000000000000000000000000000000 P 403970ae0c2e4e2da25df5a04b7b7e7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 403970ae0c2e4e2da25df5a04b7b7e7c, State: Initialized, Role: FOLLOWER
I20260927 02:38:52.499527 2858 consensus_queue.cc:260] T 00000000000000000000000000000000 P 403970ae0c2e4e2da25df5a04b7b7e7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "403970ae0c2e4e2da25df5a04b7b7e7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43123 } }
I20260927 02:38:52.499647 2858 raft_consensus.cc:388] T 00000000000000000000000000000000 P 403970ae0c2e4e2da25df5a04b7b7e7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:38:52.499691 2858 raft_consensus.cc:482] T 00000000000000000000000000000000 P 403970ae0c2e4e2da25df5a04b7b7e7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:38:52.499756 2858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 403970ae0c2e4e2da25df5a04b7b7e7c [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:38:52.500520 2858 raft_consensus.cc:504] T 00000000000000000000000000000000 P 403970ae0c2e4e2da25df5a04b7b7e7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "403970ae0c2e4e2da25df5a04b7b7e7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43123 } }
I20260927 02:38:52.500808 2858 leader_election.cc:302] T 00000000000000000000000000000000 P 403970ae0c2e4e2da25df5a04b7b7e7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 403970ae0c2e4e2da25df5a04b7b7e7c; no voters:
I20260927 02:38:52.501041 2858 leader_election.cc:288] T 00000000000000000000000000000000 P 403970ae0c2e4e2da25df5a04b7b7e7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:38:52.501169 2863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 403970ae0c2e4e2da25df5a04b7b7e7c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:38:52.501392 2863 raft_consensus.cc:686] T 00000000000000000000000000000000 P 403970ae0c2e4e2da25df5a04b7b7e7c [term 1 LEADER]: Becoming Leader. State: Replica: 403970ae0c2e4e2da25df5a04b7b7e7c, State: Running, Role: LEADER
I20260927 02:38:52.501698 2863 consensus_queue.cc:237] T 00000000000000000000000000000000 P 403970ae0c2e4e2da25df5a04b7b7e7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "403970ae0c2e4e2da25df5a04b7b7e7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43123 } }
I20260927 02:38:52.502104 2858 sys_catalog.cc:564] T 00000000000000000000000000000000 P 403970ae0c2e4e2da25df5a04b7b7e7c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:38:52.503173 2865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 403970ae0c2e4e2da25df5a04b7b7e7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 403970ae0c2e4e2da25df5a04b7b7e7c. Latest consensus state: current_term: 1 leader_uuid: "403970ae0c2e4e2da25df5a04b7b7e7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "403970ae0c2e4e2da25df5a04b7b7e7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43123 } } }
I20260927 02:38:52.503278 2865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 403970ae0c2e4e2da25df5a04b7b7e7c [sys.catalog]: This master's current role is: LEADER
I20260927 02:38:52.503530 2864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 403970ae0c2e4e2da25df5a04b7b7e7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "403970ae0c2e4e2da25df5a04b7b7e7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "403970ae0c2e4e2da25df5a04b7b7e7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43123 } } }
I20260927 02:38:52.503624 2864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 403970ae0c2e4e2da25df5a04b7b7e7c [sys.catalog]: This master's current role is: LEADER
I20260927 02:38:52.503610 2872 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:38:52.506731 2872 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:38:52.511722 2872 catalog_manager.cc:1347] Generated new cluster ID: dac5ed08a75e47788fad2948d99ea6ea
I20260927 02:38:52.511831 2872 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:38:52.517748 2872 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:38:52.518716 2872 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:38:52.525128 2872 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 403970ae0c2e4e2da25df5a04b7b7e7c: Generated new TSK 0
I20260927 02:38:52.525830 2872 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:38:52.561023 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43123
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:38:52.667718 2884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:38:52.667971 2884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:38:52.668012 2884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:38:52.671995 2884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:38:52.672052 2884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:38:52.672109 2884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:38:52.677309 2884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:38:52.678339 2884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:38:52.679330 2884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:38:52.688446 2889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:38:52.688464 2890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:38:52.688485 2892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:38:52.688697 2884 server_base.cc:1034] running on GCE node
I20260927 02:38:52.689172 2884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:38:52.689791 2884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:38:52.690941 2884 hybrid_clock.cc:648] HybridClock initialized: now 1790476732690922 us; error 30 us; skew 500 ppm
Sep 27 02:38:52 dist-test-slave-zv8z krb5kdc[2777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:38:52.693655 2884 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:38:52.694578 2884 webserver.cc:460] Webserver started at http://127.0.98.1:42617/ using document root <none> and password file <none>
I20260927 02:38:52.695086 2884 fs_manager.cc:359] Metadata directory not provided
I20260927 02:38:52.695142 2884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:38:52.695338 2884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:38:52.697315 2884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-0/data/instance:
uuid: "5fcfea564bbb4039bf9e49055e4e5eac"
format_stamp: "Formatted at 2026-09-27 02:38:52 on dist-test-slave-zv8z"
server_key: "977127f033a268e4f54e593a6d8b9168"
server_key_iv: "97ccb4dd632b83e0c07706629874f9b7"
server_key_version: "encryptionkey@0"
I20260927 02:38:52.697824 2884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5fcfea564bbb4039bf9e49055e4e5eac"
format_stamp: "Formatted at 2026-09-27 02:38:52 on dist-test-slave-zv8z"
server_key: "977127f033a268e4f54e593a6d8b9168"
server_key_iv: "97ccb4dd632b83e0c07706629874f9b7"
server_key_version: "encryptionkey@0"
I20260927 02:38:52.701575 2884 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 02:38:52.704092 2899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:52.705428 2884 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:38:52.705538 2884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-0/wal
uuid: "5fcfea564bbb4039bf9e49055e4e5eac"
format_stamp: "Formatted at 2026-09-27 02:38:52 on dist-test-slave-zv8z"
server_key: "977127f033a268e4f54e593a6d8b9168"
server_key_iv: "97ccb4dd632b83e0c07706629874f9b7"
server_key_version: "encryptionkey@0"
I20260927 02:38:52.705646 2884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:38:52.738075 2884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:38:52.740965 2884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:38:52.741127 2884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:38:52.741732 2884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:38:52.742552 2884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:38:52.742596 2884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:52.742637 2884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:38:52.742666 2884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:52.750566 2884 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37331
I20260927 02:38:52.750597 3012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37331 every 8 connection(s)
I20260927 02:38:52.751449 2884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:38:52.756909 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 2884
I20260927 02:38:52.757015 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:38:52.757263 392 external_mini_cluster.cc:1428] Setting key bd5b0dda198842cedf64731047a1bb42
Sep 27 02:38:52 dist-test-slave-zv8z krb5kdc[2777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:38:52.765702 2861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:52.753228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60005 (local address 127.0.98.62:43123)
0927 02:38:52.753364 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:52.753367 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:52.754293 (+ 926us) server_negotiation.cc:407] Connection header received
0927 02:38:52.755593 (+ 1300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:52.755596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:52.755651 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:52.755724 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:52.756938 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:52.757589 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:52.759045 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:52.759371 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:52.759470 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:52.762068 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:52.762071 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:52.762073 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:52.762097 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:52.763493 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:52.764046 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:52.764049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:52.764051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:52.764134 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:52.764505 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:52.764509 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:52.764511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:52.764697 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:52.764776 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:52.765421 (+ 645us) server_negotiation.cc:299] Negotiation successful
0927 02:38:52.765567 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:38:52.766698 3015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:52.753638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43123 (local address 127.0.98.1:60005)
0927 02:38:52.754075 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:52.754117 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:52.755392 (+ 1275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:52.755920 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:52.755928 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:52.756268 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:38:52.756771 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:52.756790 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:52.757738 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:52.757741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:52.758919 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:52.758928 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:52.759519 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:52.759522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:52.759668 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:52.760259 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:38:52.760280 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:38:52.761903 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:38:52.763604 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:52.763611 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:52.763625 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:52.763959 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:52.764222 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:52.764225 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:52.764228 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:52.764413 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:52.764801 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:38:52.764807 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:38:52.765109 (+ 302us) client_negotiation.cc:769] Sending connection context
0927 02:38:52.765350 (+ 241us) client_negotiation.cc:240] Negotiation successful
0927 02:38:52.765672 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":123,"threads_started":1}
I20260927 02:38:52.767843 3013 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43123
I20260927 02:38:52.768082 3013 heartbeater.cc:464] Registering TS with master...
I20260927 02:38:52.768563 3013 heartbeater.cc:511] Master 127.0.98.62:43123 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 02:38:52.769922 2818 ts_manager.cc:194] Registered new tserver with Master: 5fcfea564bbb4039bf9e49055e4e5eac (127.0.98.1:37331)
I20260927 02:38:52.771180 2818 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60005
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:38:52.784806 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43123
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:38:52.892279 3018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:38:52.892547 3018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:38:52.892588 3018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:38:52.896667 3018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:38:52.896727 3018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:38:52.896786 3018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:38:52.901985 3018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:38:52.903057 3018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:38:52.904076 3018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:38:52.913892 3024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:38:52.913928 3018 server_base.cc:1034] running on GCE node
W20260927 02:38:52.914027 3026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:38:52.913888 3023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:38:52.914652 3018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:38:52.915295 3018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:38:52.916464 3018 hybrid_clock.cc:648] HybridClock initialized: now 1790476732916435 us; error 41 us; skew 500 ppm
Sep 27 02:38:52 dist-test-slave-zv8z krb5kdc[2777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:38:52.919314 3018 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:38:52.920145 3018 webserver.cc:460] Webserver started at http://127.0.98.2:46173/ using document root <none> and password file <none>
I20260927 02:38:52.920703 3018 fs_manager.cc:359] Metadata directory not provided
I20260927 02:38:52.920770 3018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:38:52.920959 3018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:38:52.923055 3018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-1/data/instance:
uuid: "38319d157a0f4abc8709e4327c1e1592"
format_stamp: "Formatted at 2026-09-27 02:38:52 on dist-test-slave-zv8z"
server_key: "70d0509f9e0a0153299744b2a980723d"
server_key_iv: "26007d51236c6c9fbfc84619cdbf5c7f"
server_key_version: "encryptionkey@0"
I20260927 02:38:52.923566 3018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-1/wal/instance:
uuid: "38319d157a0f4abc8709e4327c1e1592"
format_stamp: "Formatted at 2026-09-27 02:38:52 on dist-test-slave-zv8z"
server_key: "70d0509f9e0a0153299744b2a980723d"
server_key_iv: "26007d51236c6c9fbfc84619cdbf5c7f"
server_key_version: "encryptionkey@0"
I20260927 02:38:52.927033 3018 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:38:52.929366 3033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:52.930620 3018 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 02:38:52.930732 3018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-1/wal
uuid: "38319d157a0f4abc8709e4327c1e1592"
format_stamp: "Formatted at 2026-09-27 02:38:52 on dist-test-slave-zv8z"
server_key: "70d0509f9e0a0153299744b2a980723d"
server_key_iv: "26007d51236c6c9fbfc84619cdbf5c7f"
server_key_version: "encryptionkey@0"
I20260927 02:38:52.930825 3018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:38:52.949985 3018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:38:52.953190 3018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:38:52.953367 3018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:38:52.953945 3018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:38:52.954775 3018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:38:52.954823 3018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:52.954872 3018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:38:52.954901 3018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:52.963091 3018 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37749
I20260927 02:38:52.963125 3146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37749 every 8 connection(s)
I20260927 02:38:52.963948 3018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:38:52.970386 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 3018
I20260927 02:38:52.970497 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:38:52.970762 392 external_mini_cluster.cc:1428] Setting key 5afa7ab5b4202b7903bd6e9883aa5817
Sep 27 02:38:52 dist-test-slave-zv8z krb5kdc[2777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:38:52.977671 2861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:52.965692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33667 (local address 127.0.98.62:43123)
0927 02:38:52.965847 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:52.965851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:52.966673 (+ 822us) server_negotiation.cc:407] Connection header received
0927 02:38:52.968011 (+ 1338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:52.968015 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:52.968070 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:52.968141 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:52.969254 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:52.969805 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:52.971086 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:52.971346 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:52.971453 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:52.973988 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:52.973992 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:52.973995 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:52.974021 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:52.975685 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:52.976196 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:52.976200 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:52.976202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:52.976283 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:52.976593 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:52.976596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:52.976598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:52.976794 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:52.976877 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:52.977376 (+ 499us) server_negotiation.cc:299] Negotiation successful
0927 02:38:52.977504 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 02:38:52.978489 3149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:52.966089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43123 (local address 127.0.98.2:33667)
0927 02:38:52.966537 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:52.966578 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:52.967816 (+ 1238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:52.968316 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:52.968323 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:52.968654 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:38:52.969116 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:52.969128 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:52.969917 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:52.969920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:52.970980 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:52.970990 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:52.971503 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:52.971514 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:52.971707 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:52.972285 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:38:52.972301 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:38:52.973831 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:38:52.975839 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:52.975843 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:52.975855 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:52.976093 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:52.976377 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:52.976379 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:52.976381 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:52.976508 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:52.976890 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:38:52.976897 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:38:52.977153 (+ 256us) client_negotiation.cc:769] Sending connection context
0927 02:38:52.977317 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 02:38:52.977639 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":136,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:38:52.979697 3147 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43123
I20260927 02:38:52.979920 3147 heartbeater.cc:464] Registering TS with master...
I20260927 02:38:52.980381 3147 heartbeater.cc:511] Master 127.0.98.62:43123 requested a full tablet report, sending...
I20260927 02:38:52.981359 2818 ts_manager.cc:194] Registered new tserver with Master: 38319d157a0f4abc8709e4327c1e1592 (127.0.98.2:37749)
I20260927 02:38:52.981897 2818 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33667
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 02:38:53.773495 3013 heartbeater.cc:503] Master 127.0.98.62:43123 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab.
W20260927 02:38:53.944339 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.3: real 0.972s user 0.000s sys 0.003s
I20260927 02:38:53.944813 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43123
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:38:53.983834 3147 heartbeater.cc:503] Master 127.0.98.62:43123 was elected leader, sending a full tablet report...
W20260927 02:38:54.050788 3152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:38:54.051057 3152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:38:54.051101 3152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:38:54.055104 3152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:38:54.055156 3152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:38:54.055217 3152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:38:54.060469 3152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:38:54.061497 3152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:38:54.062497 3152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:38:54.071575 3158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:38:54.071652 3152 server_base.cc:1034] running on GCE node
W20260927 02:38:54.071550 3157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:38:54.071543 3160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:38:54.072180 3152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:38:54.072749 3152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:38:54.073905 3152 hybrid_clock.cc:648] HybridClock initialized: now 1790476734073873 us; error 43 us; skew 500 ppm
Sep 27 02:38:54 dist-test-slave-zv8z krb5kdc[2777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:38:54.079483 3152 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:38:54.080379 3152 webserver.cc:460] Webserver started at http://127.0.98.3:41807/ using document root <none> and password file <none>
I20260927 02:38:54.080881 3152 fs_manager.cc:359] Metadata directory not provided
I20260927 02:38:54.080935 3152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:38:54.081089 3152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:38:54.083056 3152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-2/data/instance:
uuid: "9a271249fcbd4369ae9b4a452fc81add"
format_stamp: "Formatted at 2026-09-27 02:38:54 on dist-test-slave-zv8z"
server_key: "be9c89172bcc07e466560bc1d8784a27"
server_key_iv: "8890e104f585cd41d7bb060c4434bdfd"
server_key_version: "encryptionkey@0"
I20260927 02:38:54.083535 3152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9a271249fcbd4369ae9b4a452fc81add"
format_stamp: "Formatted at 2026-09-27 02:38:54 on dist-test-slave-zv8z"
server_key: "be9c89172bcc07e466560bc1d8784a27"
server_key_iv: "8890e104f585cd41d7bb060c4434bdfd"
server_key_version: "encryptionkey@0"
I20260927 02:38:54.087050 3152 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:38:54.089285 3167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:54.090524 3152 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:38:54.090621 3152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-2/wal
uuid: "9a271249fcbd4369ae9b4a452fc81add"
format_stamp: "Formatted at 2026-09-27 02:38:54 on dist-test-slave-zv8z"
server_key: "be9c89172bcc07e466560bc1d8784a27"
server_key_iv: "8890e104f585cd41d7bb060c4434bdfd"
server_key_version: "encryptionkey@0"
I20260927 02:38:54.090708 3152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:38:54.107563 3152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:38:54.110950 3152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:38:54.111120 3152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:38:54.111680 3152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:38:54.112469 3152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:38:54.112514 3152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:54.112564 3152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:38:54.112591 3152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:54.121452 3152 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46247
I20260927 02:38:54.121480 3280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46247 every 8 connection(s)
I20260927 02:38:54.122378 3152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:38:54.130602 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 3152
I20260927 02:38:54.130726 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:38:54.131028 392 external_mini_cluster.cc:1428] Setting key 94b6a33d01e62dce4c7c21ebf252600d
Sep 27 02:38:54 dist-test-slave-zv8z krb5kdc[2777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:38:54.136096 3283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:54.124144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49131 (local address 127.0.98.62:43123)
0927 02:38:54.124348 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:54.124352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:54.125027 (+ 675us) server_negotiation.cc:407] Connection header received
0927 02:38:54.126447 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:54.126451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:54.126509 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:54.126595 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:54.127733 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:54.128273 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:54.129351 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:54.129575 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:54.129662 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:54.132384 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:54.132388 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:54.132391 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:54.132418 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:54.133916 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:54.134542 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:54.134545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:54.134548 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:54.134640 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:54.134998 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:54.135001 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:54.135003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:54.135186 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:54.135264 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:54.135844 (+ 580us) server_negotiation.cc:299] Negotiation successful
0927 02:38:54.135969 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":56,"threads_started":1}
I20260927 02:38:54.136837 3284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:54.124456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43123 (local address 127.0.98.3:49131)
0927 02:38:54.124880 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:54.124927 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:54.126221 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:54.126770 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:54.126777 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:54.127114 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:38:54.127594 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:54.127608 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:54.128393 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:54.128396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:54.129242 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:54.129250 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:54.129752 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:54.129756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:54.129902 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:54.130559 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:38:54.130579 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:38:54.132217 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:38:54.134025 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:54.134030 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:54.134042 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:54.134425 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:54.134731 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:54.134734 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:54.134736 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:54.134878 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:54.135280 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:38:54.135285 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:38:54.135616 (+ 331us) client_negotiation.cc:769] Sending connection context
0927 02:38:54.135769 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 02:38:54.136037 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":103,"threads_started":1}
I20260927 02:38:54.137928 3281 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43123
I20260927 02:38:54.138154 3281 heartbeater.cc:464] Registering TS with master...
I20260927 02:38:54.138620 3281 heartbeater.cc:511] Master 127.0.98.62:43123 requested a full tablet report, sending...
I20260927 02:38:54.139627 2818 ts_manager.cc:194] Registered new tserver with Master: 9a271249fcbd4369ae9b4a452fc81add (127.0.98.3:46247)
I20260927 02:38:54.140175 2818 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49131
I20260927 02:38:54.144868 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 02:38:54.157009 3283 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:38:54.155583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34386 (local address 127.0.98.62:43123)
0927 02:38:54.155725 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:54.155729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:54.155867 (+ 138us) server_negotiation.cc:407] Connection header received
0927 02:38:54.156042 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:54.156044 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:54.156090 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:54.156168 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:54.156879 (+ 711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34386: BlockingRecv error: recv got EOF from 127.0.0.1:34386 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 02:38:54.158138 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 2884
I20260927 02:38:54.162946 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 3018
I20260927 02:38:54.167394 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 3152
I20260927 02:38:54.171725 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 2790
2026-09-27T02:38:54Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2930 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476718023131-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:38:54 dist-test-slave-zv8z krb5kdc[3294](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:38:54 dist-test-slave-zv8z krb5kdc[3294](info): set up 2 sockets
Sep 27 02:38:54 dist-test-slave-zv8z krb5kdc[3294](info): commencing operation
krb5kdc: starting...
W20260927 02:38:55.606505 392 mini_kdc.cc:121] Time spent starting KDC: real 1.414s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476718023131-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476718023131-392-0/krb5kdc/test-user.keytab.
Sep 27 02:38:55 dist-test-slave-zv8z krb5kdc[3294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:38:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:38:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:38:55.723165 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39423
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:38:55.827296 3307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:38:55.827550 3307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:38:55.827589 3307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:38:55.831558 3307 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:38:55.831629 3307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:38:55.831647 3307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:38:55.831661 3307 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:38:55.831676 3307 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:38:55.836843 3307 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:38:55.837875 3307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:38:55.838891 3307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:38:55.847043 3313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:38:55.847206 3307 server_base.cc:1034] running on GCE node
W20260927 02:38:55.847033 3315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:38:55.847361 3312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:38:55.847963 3307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:38:55.849043 3307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:38:55.850200 3307 hybrid_clock.cc:648] HybridClock initialized: now 1790476735850176 us; error 39 us; skew 500 ppm
Sep 27 02:38:55 dist-test-slave-zv8z krb5kdc[3294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476735, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:38:55.852895 3307 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:38:55.853729 3307 webserver.cc:460] Webserver started at http://127.0.98.62:38255/ using document root <none> and password file <none>
I20260927 02:38:55.854277 3307 fs_manager.cc:359] Metadata directory not provided
I20260927 02:38:55.854347 3307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:38:55.854514 3307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:38:55.856465 3307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476718023131-392-0/minicluster-data/master-0/data/instance:
uuid: "184eb6f763f74ddeb317fc928107d8a6"
format_stamp: "Formatted at 2026-09-27 02:38:55 on dist-test-slave-zv8z"
server_key: "a8e65a3162ff4f9adffcd6b713568648"
server_key_iv: "54164cfea069dd56c155b3018ea17e86"
server_key_version: "encryptionkey@0"
I20260927 02:38:55.856951 3307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476718023131-392-0/minicluster-data/master-0/wal/instance:
uuid: "184eb6f763f74ddeb317fc928107d8a6"
format_stamp: "Formatted at 2026-09-27 02:38:55 on dist-test-slave-zv8z"
server_key: "a8e65a3162ff4f9adffcd6b713568648"
server_key_iv: "54164cfea069dd56c155b3018ea17e86"
server_key_version: "encryptionkey@0"
I20260927 02:38:55.860699 3307 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260927 02:38:55.863010 3322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:55.864215 3307 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 02:38:55.864312 3307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "184eb6f763f74ddeb317fc928107d8a6"
format_stamp: "Formatted at 2026-09-27 02:38:55 on dist-test-slave-zv8z"
server_key: "a8e65a3162ff4f9adffcd6b713568648"
server_key_iv: "54164cfea069dd56c155b3018ea17e86"
server_key_version: "encryptionkey@0"
I20260927 02:38:55.864408 3307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:38:55.882841 3307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:38:55.886034 3307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:38:55.886226 3307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:38:55.892747 3307 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39423
I20260927 02:38:55.893011 3374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39423 every 8 connection(s)
I20260927 02:38:55.893672 3307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:38:55.896338 3375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:38:55.898584 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 3307
I20260927 02:38:55.898689 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790476718023131-392-0/minicluster-data/master-0/wal/instance
I20260927 02:38:55.898906 392 external_mini_cluster.cc:1428] Setting key 82cc701b48d565b0f5d6fc9d397cac62
I20260927 02:38:55.902347 3375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 184eb6f763f74ddeb317fc928107d8a6: Bootstrap starting.
I20260927 02:38:55.904825 3375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 184eb6f763f74ddeb317fc928107d8a6: Neither blocks nor log segments found. Creating new log.
I20260927 02:38:55.905565 3375 log.cc:824] T 00000000000000000000000000000000 P 184eb6f763f74ddeb317fc928107d8a6: Log is configured to *not* fsync() on all Append() calls
I20260927 02:38:55.907368 3375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 184eb6f763f74ddeb317fc928107d8a6: No bootstrap required, opened a new log
Sep 27 02:38:55 dist-test-slave-zv8z krb5kdc[3294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:38:55.910833 3375 raft_consensus.cc:348] T 00000000000000000000000000000000 P 184eb6f763f74ddeb317fc928107d8a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184eb6f763f74ddeb317fc928107d8a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39423 } }
I20260927 02:38:55.911096 3375 raft_consensus.cc:374] T 00000000000000000000000000000000 P 184eb6f763f74ddeb317fc928107d8a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:38:55.911145 3375 raft_consensus.cc:729] T 00000000000000000000000000000000 P 184eb6f763f74ddeb317fc928107d8a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184eb6f763f74ddeb317fc928107d8a6, State: Initialized, Role: FOLLOWER
I20260927 02:38:55.911576 3375 consensus_queue.cc:260] T 00000000000000000000000000000000 P 184eb6f763f74ddeb317fc928107d8a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184eb6f763f74ddeb317fc928107d8a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39423 } }
I20260927 02:38:55.911701 3375 raft_consensus.cc:388] T 00000000000000000000000000000000 P 184eb6f763f74ddeb317fc928107d8a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:38:55.911761 3375 raft_consensus.cc:482] T 00000000000000000000000000000000 P 184eb6f763f74ddeb317fc928107d8a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:38:55.911870 3375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 184eb6f763f74ddeb317fc928107d8a6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:38:55.912705 3375 raft_consensus.cc:504] T 00000000000000000000000000000000 P 184eb6f763f74ddeb317fc928107d8a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184eb6f763f74ddeb317fc928107d8a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39423 } }
I20260927 02:38:55.913076 3375 leader_election.cc:302] T 00000000000000000000000000000000 P 184eb6f763f74ddeb317fc928107d8a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 184eb6f763f74ddeb317fc928107d8a6; no voters:
I20260927 02:38:55.913360 3375 leader_election.cc:288] T 00000000000000000000000000000000 P 184eb6f763f74ddeb317fc928107d8a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:38:55.913440 3380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 184eb6f763f74ddeb317fc928107d8a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:38:55.913604 3380 raft_consensus.cc:686] T 00000000000000000000000000000000 P 184eb6f763f74ddeb317fc928107d8a6 [term 1 LEADER]: Becoming Leader. State: Replica: 184eb6f763f74ddeb317fc928107d8a6, State: Running, Role: LEADER
I20260927 02:38:55.913902 3378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:55.900159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51850 (local address 127.0.98.62:39423)
0927 02:38:55.900561 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:55.900580 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:55.900621 (+ 41us) server_negotiation.cc:407] Connection header received
0927 02:38:55.901375 (+ 754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:55.901398 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:55.901810 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:55.902163 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:55.902937 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:55.904407 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:55.905346 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:55.906009 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:55.906123 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:55.908395 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:55.908401 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:55.908416 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:55.908445 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:55.910982 (+ 2537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:55.911458 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:55.911463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:55.911467 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:55.911577 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:55.911896 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:55.911900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:55.911901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:55.912326 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:55.912515 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:55.912772 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 02:38:55.913038 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":240,"thread_start_us":89,"threads_started":1}
I20260927 02:38:55.913875 3380 consensus_queue.cc:237] T 00000000000000000000000000000000 P 184eb6f763f74ddeb317fc928107d8a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184eb6f763f74ddeb317fc928107d8a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39423 } }
I20260927 02:38:55.914517 3375 sys_catalog.cc:564] T 00000000000000000000000000000000 P 184eb6f763f74ddeb317fc928107d8a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:38:55.915493 3382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 184eb6f763f74ddeb317fc928107d8a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 184eb6f763f74ddeb317fc928107d8a6. Latest consensus state: current_term: 1 leader_uuid: "184eb6f763f74ddeb317fc928107d8a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184eb6f763f74ddeb317fc928107d8a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39423 } } }
I20260927 02:38:55.915501 3381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 184eb6f763f74ddeb317fc928107d8a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "184eb6f763f74ddeb317fc928107d8a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184eb6f763f74ddeb317fc928107d8a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39423 } } }
I20260927 02:38:55.915716 3381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 184eb6f763f74ddeb317fc928107d8a6 [sys.catalog]: This master's current role is: LEADER
I20260927 02:38:55.915746 3382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 184eb6f763f74ddeb317fc928107d8a6 [sys.catalog]: This master's current role is: LEADER
I20260927 02:38:55.917773 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260927 02:38:55.920966 3394 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 184eb6f763f74ddeb317fc928107d8a6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:38:55.921031 3394 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:38:55.921109 3395 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:38:55.921868 3395 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:38:55.927560 3395 catalog_manager.cc:1347] Generated new cluster ID: 2cf8b2c1a2f94bfdbba524f495e3f2a6
I20260927 02:38:55.927625 3395 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:38:55.936118 3378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:55.930823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51860 (local address 127.0.98.62:39423)
0927 02:38:55.930938 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:55.930943 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:55.931137 (+ 194us) server_negotiation.cc:407] Connection header received
0927 02:38:55.931248 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:55.931252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:55.931299 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:55.931393 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:55.931914 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:55.932760 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:55.933846 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:55.934199 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:55.934334 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:55.935173 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:55.935177 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:55.935180 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:38:55.935718 (+ 538us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:55.935751 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:38:55.935770 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:55.935866 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:55.935919 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 02:38:55.935977 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 02:38:55.952581 3395 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:38:55.953848 3395 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:38:55.960734 3395 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 184eb6f763f74ddeb317fc928107d8a6: Generated new TSK 0
I20260927 02:38:55.961513 3395 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:38:55.969312 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 3307
2026-09-27T02:38:55Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1795 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:38:56 dist-test-slave-zv8z krb5kdc[3408](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:38:56 dist-test-slave-zv8z krb5kdc[3408](info): set up 2 sockets
Sep 27 02:38:56 dist-test-slave-zv8z krb5kdc[3408](info): commencing operation
krb5kdc: starting...
W20260927 02:38:56.921983 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/krb5kdc/test-user.keytab.
Sep 27 02:38:56 dist-test-slave-zv8z krb5kdc[3408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:38:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:38:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:38:57.033119 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43479
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43479
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:38:57.137588 3421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:38:57.137867 3421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:38:57.137923 3421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:38:57.141852 3421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:38:57.141908 3421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:38:57.141935 3421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:38:57.141955 3421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:38:57.141983 3421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:38:57.147109 3421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:38:57.148109 3421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:38:57.149092 3421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:38:57.157403 3426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:38:57.157454 3421 server_base.cc:1034] running on GCE node
W20260927 02:38:57.157462 3427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:38:57.157650 3429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:38:57.158185 3421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:38:57.159297 3421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:38:57.160478 3421 hybrid_clock.cc:648] HybridClock initialized: now 1790476737160455 us; error 34 us; skew 500 ppm
Sep 27 02:38:57 dist-test-slave-zv8z krb5kdc[3408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:38:57.163236 3421 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:38:57.164124 3421 webserver.cc:460] Webserver started at http://127.0.98.62:37617/ using document root <none> and password file <none>
I20260927 02:38:57.164642 3421 fs_manager.cc:359] Metadata directory not provided
I20260927 02:38:57.164706 3421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:38:57.164899 3421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:38:57.166900 3421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/master-0/data/instance:
uuid: "72d588212c6d4135bf4054d9355bd44f"
format_stamp: "Formatted at 2026-09-27 02:38:57 on dist-test-slave-zv8z"
server_key: "b1e07424c0bc2808a4b5cd73104e9f5c"
server_key_iv: "e546daf74bbb66cd0da886888f4667e1"
server_key_version: "encryptionkey@0"
I20260927 02:38:57.167418 3421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/master-0/wal/instance:
uuid: "72d588212c6d4135bf4054d9355bd44f"
format_stamp: "Formatted at 2026-09-27 02:38:57 on dist-test-slave-zv8z"
server_key: "b1e07424c0bc2808a4b5cd73104e9f5c"
server_key_iv: "e546daf74bbb66cd0da886888f4667e1"
server_key_version: "encryptionkey@0"
I20260927 02:38:57.170897 3421 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:38:57.173223 3436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:57.174427 3421 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:38:57.174533 3421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "72d588212c6d4135bf4054d9355bd44f"
format_stamp: "Formatted at 2026-09-27 02:38:57 on dist-test-slave-zv8z"
server_key: "b1e07424c0bc2808a4b5cd73104e9f5c"
server_key_iv: "e546daf74bbb66cd0da886888f4667e1"
server_key_version: "encryptionkey@0"
I20260927 02:38:57.174624 3421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:38:57.191537 3421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:38:57.194694 3421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:38:57.194882 3421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:38:57.201711 3421 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43479
I20260927 02:38:57.202013 3488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43479 every 8 connection(s)
I20260927 02:38:57.202729 3421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:38:57.205444 3489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:38:57.208434 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 3421
I20260927 02:38:57.208555 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/master-0/wal/instance
I20260927 02:38:57.208858 392 external_mini_cluster.cc:1428] Setting key 9bca5e0eea9602228e9fe7593a64b576
I20260927 02:38:57.211259 3489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72d588212c6d4135bf4054d9355bd44f: Bootstrap starting.
I20260927 02:38:57.213407 3489 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72d588212c6d4135bf4054d9355bd44f: Neither blocks nor log segments found. Creating new log.
I20260927 02:38:57.214131 3489 log.cc:824] T 00000000000000000000000000000000 P 72d588212c6d4135bf4054d9355bd44f: Log is configured to *not* fsync() on all Append() calls
I20260927 02:38:57.216053 3489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72d588212c6d4135bf4054d9355bd44f: No bootstrap required, opened a new log
Sep 27 02:38:57 dist-test-slave-zv8z krb5kdc[3408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:38:57.220055 3489 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72d588212c6d4135bf4054d9355bd44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72d588212c6d4135bf4054d9355bd44f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43479 } }
I20260927 02:38:57.220270 3489 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72d588212c6d4135bf4054d9355bd44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:38:57.220304 3489 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72d588212c6d4135bf4054d9355bd44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72d588212c6d4135bf4054d9355bd44f, State: Initialized, Role: FOLLOWER
I20260927 02:38:57.220764 3489 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72d588212c6d4135bf4054d9355bd44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72d588212c6d4135bf4054d9355bd44f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43479 } }
I20260927 02:38:57.220899 3489 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72d588212c6d4135bf4054d9355bd44f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:38:57.220963 3489 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d588212c6d4135bf4054d9355bd44f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:38:57.221058 3489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72d588212c6d4135bf4054d9355bd44f [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:38:57.221882 3489 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d588212c6d4135bf4054d9355bd44f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72d588212c6d4135bf4054d9355bd44f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43479 } }
I20260927 02:38:57.222221 3489 leader_election.cc:302] T 00000000000000000000000000000000 P 72d588212c6d4135bf4054d9355bd44f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72d588212c6d4135bf4054d9355bd44f; no voters:
I20260927 02:38:57.222555 3489 leader_election.cc:288] T 00000000000000000000000000000000 P 72d588212c6d4135bf4054d9355bd44f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:38:57.222653 3494 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72d588212c6d4135bf4054d9355bd44f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:38:57.222808 3494 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72d588212c6d4135bf4054d9355bd44f [term 1 LEADER]: Becoming Leader. State: Replica: 72d588212c6d4135bf4054d9355bd44f, State: Running, Role: LEADER
I20260927 02:38:57.223150 3494 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72d588212c6d4135bf4054d9355bd44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72d588212c6d4135bf4054d9355bd44f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43479 } }
I20260927 02:38:57.223222 3492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:57.210223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34582 (local address 127.0.98.62:43479)
0927 02:38:57.210729 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:57.210744 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:57.210781 (+ 37us) server_negotiation.cc:407] Connection header received
0927 02:38:57.211513 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:57.211533 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:57.211878 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:57.212193 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:57.213001 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:57.214044 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:57.215114 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:57.215583 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:57.215726 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:57.218508 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:57.218513 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:57.218525 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:57.218552 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:57.220251 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:57.220684 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:57.220689 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:57.220693 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:57.220804 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:57.221139 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:57.221143 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:57.221145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:57.221539 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:57.221738 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:57.222013 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 02:38:57.222324 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":139,"threads_started":1}
I20260927 02:38:57.223810 3489 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72d588212c6d4135bf4054d9355bd44f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:38:57.226083 3496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72d588212c6d4135bf4054d9355bd44f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72d588212c6d4135bf4054d9355bd44f. Latest consensus state: current_term: 1 leader_uuid: "72d588212c6d4135bf4054d9355bd44f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72d588212c6d4135bf4054d9355bd44f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43479 } } }
I20260927 02:38:57.226188 3496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72d588212c6d4135bf4054d9355bd44f [sys.catalog]: This master's current role is: LEADER
I20260927 02:38:57.227221 3495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72d588212c6d4135bf4054d9355bd44f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72d588212c6d4135bf4054d9355bd44f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72d588212c6d4135bf4054d9355bd44f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43479 } } }
I20260927 02:38:57.227384 3495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72d588212c6d4135bf4054d9355bd44f [sys.catalog]: This master's current role is: LEADER
W20260927 02:38:57.228163 3508 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 72d588212c6d4135bf4054d9355bd44f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:38:57.228224 3508 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:38:57.228291 3509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:38:57.228909 3509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:38:57.234370 3509 catalog_manager.cc:1347] Generated new cluster ID: 6eb4ebde280446fa825a748fcaf3c6d9
I20260927 02:38:57.234431 3509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:38:57.260589 3509 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:38:57.261669 3509 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:38:57.272310 3509 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72d588212c6d4135bf4054d9355bd44f: Generated new TSK 0
I20260927 02:38:57.272965 3509 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:38:57.305580 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43479
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:38:57.412537 3515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:38:57.412794 3515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:38:57.412837 3515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:38:57.416860 3515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:38:57.416920 3515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:38:57.416978 3515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:38:57.422111 3515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:38:57.423106 3515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:38:57.424090 3515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:38:57.433218 3521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:38:57.433211 3523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:38:57.433252 3515 server_base.cc:1034] running on GCE node
W20260927 02:38:57.433218 3520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:38:57.433777 3515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:38:57.434417 3515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:38:57.435600 3515 hybrid_clock.cc:648] HybridClock initialized: now 1790476737435584 us; error 25 us; skew 500 ppm
Sep 27 02:38:57 dist-test-slave-zv8z krb5kdc[3408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:38:57.438309 3515 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:38:57.439126 3515 webserver.cc:460] Webserver started at http://127.0.98.1:33155/ using document root <none> and password file <none>
I20260927 02:38:57.439618 3515 fs_manager.cc:359] Metadata directory not provided
I20260927 02:38:57.439673 3515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:38:57.439841 3515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:38:57.441751 3515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-0/data/instance:
uuid: "c35633f82a914c8ea3ce0621e037f261"
format_stamp: "Formatted at 2026-09-27 02:38:57 on dist-test-slave-zv8z"
server_key: "035aa3c39466f21404ac0beacba52cf8"
server_key_iv: "cc40c652d430efba7dd2a3e2686adbc5"
server_key_version: "encryptionkey@0"
I20260927 02:38:57.442235 3515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c35633f82a914c8ea3ce0621e037f261"
format_stamp: "Formatted at 2026-09-27 02:38:57 on dist-test-slave-zv8z"
server_key: "035aa3c39466f21404ac0beacba52cf8"
server_key_iv: "cc40c652d430efba7dd2a3e2686adbc5"
server_key_version: "encryptionkey@0"
I20260927 02:38:57.445739 3515 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 02:38:57.448436 3530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:57.449645 3515 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:38:57.449752 3515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-0/wal
uuid: "c35633f82a914c8ea3ce0621e037f261"
format_stamp: "Formatted at 2026-09-27 02:38:57 on dist-test-slave-zv8z"
server_key: "035aa3c39466f21404ac0beacba52cf8"
server_key_iv: "cc40c652d430efba7dd2a3e2686adbc5"
server_key_version: "encryptionkey@0"
I20260927 02:38:57.449856 3515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:38:57.469717 3515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:38:57.472777 3515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:38:57.472957 3515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:38:57.473526 3515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:38:57.474412 3515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:38:57.474458 3515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:57.474501 3515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:38:57.474529 3515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:57.483026 3515 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42719
I20260927 02:38:57.483057 3643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42719 every 8 connection(s)
I20260927 02:38:57.483858 3515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:38:57.491238 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 3515
I20260927 02:38:57.491367 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:38:57.491624 392 external_mini_cluster.cc:1428] Setting key 297089e9be4cd83e2e8621c0e18f06d2
Sep 27 02:38:57 dist-test-slave-zv8z krb5kdc[3408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:38:57.497295 3492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:57.485548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47239 (local address 127.0.98.62:43479)
0927 02:38:57.485674 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:57.485678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:57.486497 (+ 819us) server_negotiation.cc:407] Connection header received
0927 02:38:57.487859 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:57.487863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:57.487929 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:57.488000 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:57.489142 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:57.489669 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:57.490823 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:57.491090 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:57.491198 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:57.493775 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:57.493778 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:57.493781 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:57.493806 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:57.495199 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:57.495688 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:57.495691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:57.495692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:57.495806 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:57.496182 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:57.496185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:57.496186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:57.496391 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:57.496475 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:57.497067 (+ 592us) server_negotiation.cc:299] Negotiation successful
0927 02:38:57.497173 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 02:38:57.498245 3646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:57.485909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43479 (local address 127.0.98.1:47239)
0927 02:38:57.486355 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:57.486405 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:57.487652 (+ 1247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:57.488182 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:57.488189 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:57.488546 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:38:57.489003 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:57.489017 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:57.489782 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:57.489786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:57.490709 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:57.490718 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:57.491243 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:57.491247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:57.491431 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:57.492012 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:38:57.492027 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:38:57.493615 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:38:57.495307 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:57.495312 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:57.495328 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:57.495579 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:57.495893 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:57.495896 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:57.495898 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:57.496069 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:57.496487 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:38:57.496493 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:38:57.496795 (+ 302us) client_negotiation.cc:769] Sending connection context
0927 02:38:57.497013 (+ 218us) client_negotiation.cc:240] Negotiation successful
0927 02:38:57.497410 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":99,"threads_started":1}
I20260927 02:38:57.499612 3644 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43479
I20260927 02:38:57.499828 3644 heartbeater.cc:464] Registering TS with master...
I20260927 02:38:57.500288 3644 heartbeater.cc:511] Master 127.0.98.62:43479 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:38:57.501732 3450 ts_manager.cc:194] Registered new tserver with Master: c35633f82a914c8ea3ce0621e037f261 (127.0.98.1:42719)
I20260927 02:38:57.502949 3450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47239
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:38:57.517261 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43479
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:38:57.623721 3649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:38:57.623967 3649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:38:57.624012 3649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:38:57.628101 3649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:38:57.628157 3649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:38:57.628212 3649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:38:57.633160 3649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:38:57.634074 3649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:38:57.635004 3649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:38:57.643862 3654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:38:57.643854 3657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:38:57.643885 3655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:38:57.644114 3649 server_base.cc:1034] running on GCE node
I20260927 02:38:57.644549 3649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:38:57.645148 3649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:38:57.646337 3649 hybrid_clock.cc:648] HybridClock initialized: now 1790476737646308 us; error 41 us; skew 500 ppm
Sep 27 02:38:57 dist-test-slave-zv8z krb5kdc[3408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:38:57.648831 3649 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:38:57.649628 3649 webserver.cc:460] Webserver started at http://127.0.98.2:36147/ using document root <none> and password file <none>
I20260927 02:38:57.650106 3649 fs_manager.cc:359] Metadata directory not provided
I20260927 02:38:57.650158 3649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:38:57.650372 3649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:38:57.652313 3649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-1/data/instance:
uuid: "a25a3406777a453b9198601f214ecc7f"
format_stamp: "Formatted at 2026-09-27 02:38:57 on dist-test-slave-zv8z"
server_key: "7c0adf30d45b6bd0d8a7f5a0f632793e"
server_key_iv: "306c568ab10cd938d7b606d2a0a0aa92"
server_key_version: "encryptionkey@0"
I20260927 02:38:57.652820 3649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a25a3406777a453b9198601f214ecc7f"
format_stamp: "Formatted at 2026-09-27 02:38:57 on dist-test-slave-zv8z"
server_key: "7c0adf30d45b6bd0d8a7f5a0f632793e"
server_key_iv: "306c568ab10cd938d7b606d2a0a0aa92"
server_key_version: "encryptionkey@0"
I20260927 02:38:57.656790 3649 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 02:38:57.659811 3664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:57.661362 3649 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 02:38:57.661479 3649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-1/wal
uuid: "a25a3406777a453b9198601f214ecc7f"
format_stamp: "Formatted at 2026-09-27 02:38:57 on dist-test-slave-zv8z"
server_key: "7c0adf30d45b6bd0d8a7f5a0f632793e"
server_key_iv: "306c568ab10cd938d7b606d2a0a0aa92"
server_key_version: "encryptionkey@0"
I20260927 02:38:57.661590 3649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:38:57.676978 3649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:38:57.679991 3649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:38:57.680135 3649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:38:57.680649 3649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:38:57.681407 3649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:38:57.681452 3649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:57.681488 3649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:38:57.681507 3649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:57.689662 3649 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33487
I20260927 02:38:57.689689 3777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33487 every 8 connection(s)
I20260927 02:38:57.690577 3649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:38:57.692916 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 3649
I20260927 02:38:57.693045 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:38:57.693358 392 external_mini_cluster.cc:1428] Setting key 5620f51afe7141faf28ddf8adc185314
Sep 27 02:38:57 dist-test-slave-zv8z krb5kdc[3408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 02:38:57.705687 3492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:57.692340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41393 (local address 127.0.98.62:43479)
0927 02:38:57.692484 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:57.692488 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:57.693521 (+ 1033us) server_negotiation.cc:407] Connection header received
0927 02:38:57.695159 (+ 1638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:57.695162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:57.695205 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:57.695276 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:57.696788 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:57.697567 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:57.699220 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:57.699452 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:57.699533 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:57.702127 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:57.702129 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:57.702132 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:57.702156 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:57.703602 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:57.704137 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:57.704140 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:57.704142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:57.704227 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:57.704616 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:57.704619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:57.704620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:57.704806 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:57.704951 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:57.705403 (+ 452us) server_negotiation.cc:299] Negotiation successful
0927 02:38:57.705523 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 02:38:57.706595 3780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:57.692672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43479 (local address 127.0.98.2:41393)
0927 02:38:57.693288 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:57.693373 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:57.694929 (+ 1556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:57.695490 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:57.695499 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:57.695971 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:38:57.696638 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:57.696657 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:57.697712 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:57.697716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:57.699090 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:57.699103 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:57.699600 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:57.699605 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:57.699809 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:57.700399 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:38:57.700420 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:38:57.701967 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:38:57.703741 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:57.703745 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:57.703758 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:57.704025 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:57.704352 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:57.704355 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:57.704357 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:57.704519 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:57.704913 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:38:57.704920 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:38:57.705197 (+ 277us) client_negotiation.cc:769] Sending connection context
0927 02:38:57.705359 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 02:38:57.705650 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":104,"threads_started":1}
I20260927 02:38:57.707685 3778 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43479
I20260927 02:38:57.707907 3778 heartbeater.cc:464] Registering TS with master...
I20260927 02:38:57.708366 3778 heartbeater.cc:511] Master 127.0.98.62:43479 requested a full tablet report, sending...
I20260927 02:38:57.709316 3450 ts_manager.cc:194] Registered new tserver with Master: a25a3406777a453b9198601f214ecc7f (127.0.98.2:33487)
I20260927 02:38:57.709851 3450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41393
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:38:57.721020 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43479
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:38:57.825034 3783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:38:57.825285 3783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:38:57.825333 3783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:38:57.829351 3783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:38:57.829404 3783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:38:57.829461 3783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:38:57.834805 3783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:38:57.835825 3783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:38:57.836786 3783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:38:57.845976 3789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:38:57.845983 3788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:38:57.846165 3783 server_base.cc:1034] running on GCE node
W20260927 02:38:57.845996 3791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:38:57.846668 3783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:38:57.847255 3783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:38:57.848420 3783 hybrid_clock.cc:648] HybridClock initialized: now 1790476737848392 us; error 39 us; skew 500 ppm
Sep 27 02:38:57 dist-test-slave-zv8z krb5kdc[3408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:38:57.851015 3783 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:38:57.851855 3783 webserver.cc:460] Webserver started at http://127.0.98.3:38591/ using document root <none> and password file <none>
I20260927 02:38:57.852380 3783 fs_manager.cc:359] Metadata directory not provided
I20260927 02:38:57.852469 3783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:38:57.852670 3783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:38:57.855165 3783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-2/data/instance:
uuid: "aa13de7e39e44e278647d7e6f4147a6d"
format_stamp: "Formatted at 2026-09-27 02:38:57 on dist-test-slave-zv8z"
server_key: "27c2a8f0f5f67cb3561690670d262c20"
server_key_iv: "b0292146cf5000c22b1a57a02071793e"
server_key_version: "encryptionkey@0"
I20260927 02:38:57.855674 3783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-2/wal/instance:
uuid: "aa13de7e39e44e278647d7e6f4147a6d"
format_stamp: "Formatted at 2026-09-27 02:38:57 on dist-test-slave-zv8z"
server_key: "27c2a8f0f5f67cb3561690670d262c20"
server_key_iv: "b0292146cf5000c22b1a57a02071793e"
server_key_version: "encryptionkey@0"
I20260927 02:38:57.859124 3783 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 02:38:57.861382 3798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:57.862535 3783 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:38:57.862630 3783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-2/wal
uuid: "aa13de7e39e44e278647d7e6f4147a6d"
format_stamp: "Formatted at 2026-09-27 02:38:57 on dist-test-slave-zv8z"
server_key: "27c2a8f0f5f67cb3561690670d262c20"
server_key_iv: "b0292146cf5000c22b1a57a02071793e"
server_key_version: "encryptionkey@0"
I20260927 02:38:57.862718 3783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:38:57.876060 3783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:38:57.879184 3783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:38:57.879343 3783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:38:57.879885 3783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:38:57.880647 3783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:38:57.880689 3783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:57.880730 3783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:38:57.880759 3783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:57.888720 3783 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46295
I20260927 02:38:57.888759 3911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46295 every 8 connection(s)
I20260927 02:38:57.889564 3783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:38:57.896627 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 3783
I20260927 02:38:57.896730 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790476718023131-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:38:57.897027 392 external_mini_cluster.cc:1428] Setting key 0de882dadfdc56997c3cba4d270c060a
Sep 27 02:38:57 dist-test-slave-zv8z krb5kdc[3408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:38:57.902530 3492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:57.891313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57787 (local address 127.0.98.62:43479)
0927 02:38:57.891480 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:57.891485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:57.892248 (+ 763us) server_negotiation.cc:407] Connection header received
0927 02:38:57.893535 (+ 1287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:57.893537 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:57.893583 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:57.893654 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:57.894786 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:57.895430 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:57.896488 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:57.896787 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:57.896890 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:57.899451 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:57.899454 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:57.899456 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:57.899479 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:57.900703 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:57.901153 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:57.901157 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:57.901158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:57.901227 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:57.901521 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:57.901524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:57.901526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:57.901690 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:57.901769 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:57.902228 (+ 459us) server_negotiation.cc:299] Negotiation successful
0927 02:38:57.902366 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 02:38:57.903259 3914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:57.891693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43479 (local address 127.0.98.3:57787)
0927 02:38:57.892107 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:57.892147 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:57.893330 (+ 1183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:57.893803 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:57.893809 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:57.894150 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:38:57.894657 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:57.894671 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:57.895527 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:57.895529 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:57.896374 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:57.896382 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:57.896946 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:57.896949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:57.897180 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:57.897690 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:38:57.897709 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:38:57.899299 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:38:57.900805 (+ 1506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:57.900811 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:57.900825 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:57.901065 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:57.901304 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:57.901307 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:57.901310 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:57.901439 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:57.901782 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:38:57.901788 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:38:57.902019 (+ 231us) client_negotiation.cc:769] Sending connection context
0927 02:38:57.902176 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 02:38:57.902500 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":95,"threads_started":1}
I20260927 02:38:57.904215 3912 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43479
I20260927 02:38:57.904408 3912 heartbeater.cc:464] Registering TS with master...
I20260927 02:38:57.904817 3912 heartbeater.cc:511] Master 127.0.98.62:43479 requested a full tablet report, sending...
I20260927 02:38:57.905682 3450 ts_manager.cc:194] Registered new tserver with Master: aa13de7e39e44e278647d7e6f4147a6d (127.0.98.3:46295)
I20260927 02:38:57.906183 3450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57787
I20260927 02:38:57.910482 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 02:38:57 dist-test-slave-zv8z krb5kdc[3408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476737, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 02:38:57 dist-test-slave-zv8z krb5kdc[3408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476737, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:38:57.933329 3492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:57.925689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34586 (local address 127.0.98.62:43479)
0927 02:38:57.925823 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:57.925827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:57.925955 (+ 128us) server_negotiation.cc:407] Connection header received
0927 02:38:57.926081 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:57.926084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:57.926125 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:57.926198 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:57.926860 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:57.927372 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:57.928169 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:57.928369 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:57.928439 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:57.930638 (+ 2199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:57.930641 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:57.930643 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:57.930665 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:57.931977 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:57.932331 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:57.932334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:57.932335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:57.932416 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:57.932609 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:57.932611 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:57.932612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:57.932849 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:57.932949 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:57.933128 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 02:38:57.933212 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260927 02:38:57.933681 3450 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:34586
I20260927 02:38:57.934450 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 3515
I20260927 02:38:57.939091 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 3649
I20260927 02:38:57.943372 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 3783
I20260927 02:38:57.947584 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 3421
2026-09-27T02:38:57Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1979 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:38:57 dist-test-slave-zv8z krb5kdc[3924](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:38:57 dist-test-slave-zv8z krb5kdc[3924](info): set up 2 sockets
Sep 27 02:38:57 dist-test-slave-zv8z krb5kdc[3924](info): commencing operation
krb5kdc: starting...
W20260927 02:38:58.898041 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/krb5kdc/test-user.keytab.
Sep 27 02:38:58 dist-test-slave-zv8z krb5kdc[3924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:38:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:38:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:38:59.010511 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35927
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35927
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:38:59.123237 3937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:38:59.123581 3937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:38:59.123694 3937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:38:59.129372 3937 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:38:59.129448 3937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:38:59.129475 3937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:38:59.129498 3937 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:38:59.129525 3937 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:38:59.134922 3937 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:38:59.135982 3937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:38:59.137017 3937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:38:59.145282 3943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:38:59.145445 3937 server_base.cc:1034] running on GCE node
W20260927 02:38:59.145287 3945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:38:59.145287 3942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:38:59.146155 3937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:38:59.147298 3937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:38:59.148458 3937 hybrid_clock.cc:648] HybridClock initialized: now 1790476739148433 us; error 38 us; skew 500 ppm
Sep 27 02:38:59 dist-test-slave-zv8z krb5kdc[3924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:38:59.151216 3937 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:38:59.152079 3937 webserver.cc:460] Webserver started at http://127.0.98.62:35929/ using document root <none> and password file <none>
I20260927 02:38:59.152601 3937 fs_manager.cc:359] Metadata directory not provided
I20260927 02:38:59.152661 3937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:38:59.152853 3937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:38:59.154937 3937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/master-0/data/instance:
uuid: "f9a1fe5f96d8471380b781f778bfdc99"
format_stamp: "Formatted at 2026-09-27 02:38:59 on dist-test-slave-zv8z"
server_key: "8093322c9cd9067c015a477fad9702f8"
server_key_iv: "b623c9392730cea486fa526806c51ada"
server_key_version: "encryptionkey@0"
I20260927 02:38:59.155534 3937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/master-0/wal/instance:
uuid: "f9a1fe5f96d8471380b781f778bfdc99"
format_stamp: "Formatted at 2026-09-27 02:38:59 on dist-test-slave-zv8z"
server_key: "8093322c9cd9067c015a477fad9702f8"
server_key_iv: "b623c9392730cea486fa526806c51ada"
server_key_version: "encryptionkey@0"
I20260927 02:38:59.159171 3937 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:38:59.161571 3952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:59.162844 3937 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:38:59.162958 3937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "f9a1fe5f96d8471380b781f778bfdc99"
format_stamp: "Formatted at 2026-09-27 02:38:59 on dist-test-slave-zv8z"
server_key: "8093322c9cd9067c015a477fad9702f8"
server_key_iv: "b623c9392730cea486fa526806c51ada"
server_key_version: "encryptionkey@0"
I20260927 02:38:59.163049 3937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:38:59.191749 3937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:38:59.194968 3937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:38:59.195160 3937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:38:59.201890 3937 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35927
I20260927 02:38:59.201900 4004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35927 every 8 connection(s)
I20260927 02:38:59.202841 3937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:38:59.205641 4005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:38:59.206322 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 3937
I20260927 02:38:59.206439 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/master-0/wal/instance
I20260927 02:38:59.206694 392 external_mini_cluster.cc:1428] Setting key aab91806b6f32c562b706d5587bd28d2
I20260927 02:38:59.212142 4005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9a1fe5f96d8471380b781f778bfdc99: Bootstrap starting.
I20260927 02:38:59.214639 4005 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9a1fe5f96d8471380b781f778bfdc99: Neither blocks nor log segments found. Creating new log.
I20260927 02:38:59.215348 4005 log.cc:824] T 00000000000000000000000000000000 P f9a1fe5f96d8471380b781f778bfdc99: Log is configured to *not* fsync() on all Append() calls
Sep 27 02:38:59 dist-test-slave-zv8z krb5kdc[3924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:38:59.217310 4005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9a1fe5f96d8471380b781f778bfdc99: No bootstrap required, opened a new log
I20260927 02:38:59.220716 4008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:59.208027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44476 (local address 127.0.98.62:35927)
0927 02:38:59.208440 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:59.208460 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:59.208505 (+ 45us) server_negotiation.cc:407] Connection header received
0927 02:38:59.209311 (+ 806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:59.209334 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:59.209754 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:59.210106 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:59.210799 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:59.212229 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:59.213064 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:59.213764 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:59.213912 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:59.216403 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:59.216408 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:59.216422 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:59.216448 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:59.218096 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:59.218547 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:59.218551 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:59.218556 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:59.218663 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:59.218918 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:59.218921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:59.218922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:59.219290 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:59.219481 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:59.219737 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 02:38:59.219997 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":102,"threads_started":1}
I20260927 02:38:59.220732 4005 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9a1fe5f96d8471380b781f778bfdc99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9a1fe5f96d8471380b781f778bfdc99" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35927 } }
I20260927 02:38:59.220988 4005 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9a1fe5f96d8471380b781f778bfdc99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:38:59.221043 4005 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9a1fe5f96d8471380b781f778bfdc99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9a1fe5f96d8471380b781f778bfdc99, State: Initialized, Role: FOLLOWER
I20260927 02:38:59.221473 4005 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9a1fe5f96d8471380b781f778bfdc99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9a1fe5f96d8471380b781f778bfdc99" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35927 } }
I20260927 02:38:59.221604 4005 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9a1fe5f96d8471380b781f778bfdc99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:38:59.221686 4005 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9a1fe5f96d8471380b781f778bfdc99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:38:59.221784 4005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9a1fe5f96d8471380b781f778bfdc99 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:38:59.222693 4005 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9a1fe5f96d8471380b781f778bfdc99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9a1fe5f96d8471380b781f778bfdc99" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35927 } }
I20260927 02:38:59.223028 4005 leader_election.cc:302] T 00000000000000000000000000000000 P f9a1fe5f96d8471380b781f778bfdc99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9a1fe5f96d8471380b781f778bfdc99; no voters:
I20260927 02:38:59.223310 4005 leader_election.cc:288] T 00000000000000000000000000000000 P f9a1fe5f96d8471380b781f778bfdc99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:38:59.223435 4010 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9a1fe5f96d8471380b781f778bfdc99 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:38:59.223644 4010 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9a1fe5f96d8471380b781f778bfdc99 [term 1 LEADER]: Becoming Leader. State: Replica: f9a1fe5f96d8471380b781f778bfdc99, State: Running, Role: LEADER
I20260927 02:38:59.223946 4010 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9a1fe5f96d8471380b781f778bfdc99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9a1fe5f96d8471380b781f778bfdc99" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35927 } }
I20260927 02:38:59.224396 4005 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9a1fe5f96d8471380b781f778bfdc99 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:38:59.225436 4012 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9a1fe5f96d8471380b781f778bfdc99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9a1fe5f96d8471380b781f778bfdc99. Latest consensus state: current_term: 1 leader_uuid: "f9a1fe5f96d8471380b781f778bfdc99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9a1fe5f96d8471380b781f778bfdc99" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35927 } } }
I20260927 02:38:59.225562 4012 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9a1fe5f96d8471380b781f778bfdc99 [sys.catalog]: This master's current role is: LEADER
I20260927 02:38:59.225919 4019 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:38:59.225885 4011 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9a1fe5f96d8471380b781f778bfdc99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9a1fe5f96d8471380b781f778bfdc99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9a1fe5f96d8471380b781f778bfdc99" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35927 } } }
I20260927 02:38:59.226025 4011 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9a1fe5f96d8471380b781f778bfdc99 [sys.catalog]: This master's current role is: LEADER
I20260927 02:38:59.228742 4019 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:38:59.233911 4019 catalog_manager.cc:1347] Generated new cluster ID: d5df5d3101bc41d9a7c2b7ef5bc15cd1
I20260927 02:38:59.233976 4019 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:38:59.261539 4019 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:38:59.262590 4019 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:38:59.270656 4019 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9a1fe5f96d8471380b781f778bfdc99: Generated new TSK 0
I20260927 02:38:59.271322 4019 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:38:59.306442 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35927
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:38:59.417060 4031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:38:59.417315 4031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:38:59.417356 4031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:38:59.421317 4031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:38:59.421373 4031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:38:59.421430 4031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:38:59.426636 4031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:38:59.427593 4031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:38:59.428529 4031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:38:59.437620 4037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:38:59.437641 4036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:38:59.437618 4039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:38:59.437937 4031 server_base.cc:1034] running on GCE node
I20260927 02:38:59.438383 4031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:38:59.438992 4031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:38:59.440166 4031 hybrid_clock.cc:648] HybridClock initialized: now 1790476739440137 us; error 39 us; skew 500 ppm
Sep 27 02:38:59 dist-test-slave-zv8z krb5kdc[3924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:38:59.442869 4031 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:38:59.443739 4031 webserver.cc:460] Webserver started at http://127.0.98.1:33025/ using document root <none> and password file <none>
I20260927 02:38:59.444270 4031 fs_manager.cc:359] Metadata directory not provided
I20260927 02:38:59.444331 4031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:38:59.444514 4031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:38:59.446579 4031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/ts-0/data/instance:
uuid: "38c11c8adb28462ab4a89c9a423b1ea4"
format_stamp: "Formatted at 2026-09-27 02:38:59 on dist-test-slave-zv8z"
server_key: "dbebd3f4a56957897568870b039798cd"
server_key_iv: "812ba575fe112f2ba3075dfbe7c2ff63"
server_key_version: "encryptionkey@0"
I20260927 02:38:59.447127 4031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/ts-0/wal/instance:
uuid: "38c11c8adb28462ab4a89c9a423b1ea4"
format_stamp: "Formatted at 2026-09-27 02:38:59 on dist-test-slave-zv8z"
server_key: "dbebd3f4a56957897568870b039798cd"
server_key_iv: "812ba575fe112f2ba3075dfbe7c2ff63"
server_key_version: "encryptionkey@0"
I20260927 02:38:59.450771 4031 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:38:59.453173 4046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:59.454656 4031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:38:59.454762 4031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/ts-0/wal
uuid: "38c11c8adb28462ab4a89c9a423b1ea4"
format_stamp: "Formatted at 2026-09-27 02:38:59 on dist-test-slave-zv8z"
server_key: "dbebd3f4a56957897568870b039798cd"
server_key_iv: "812ba575fe112f2ba3075dfbe7c2ff63"
server_key_version: "encryptionkey@0"
I20260927 02:38:59.454862 4031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:38:59.466459 4031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:38:59.469480 4031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:38:59.469622 4031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:38:59.470134 4031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:38:59.470923 4031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:38:59.470975 4031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:59.471011 4031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:38:59.471032 4031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:59.479467 4031 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42539
I20260927 02:38:59.479488 4159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42539 every 8 connection(s)
I20260927 02:38:59.480369 4031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:38:59.482592 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 4031
I20260927 02:38:59.482703 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:38:59.483037 392 external_mini_cluster.cc:1428] Setting key f1c1f9de8f437da35f42ad2129bdb2e7
Sep 27 02:38:59 dist-test-slave-zv8z krb5kdc[3924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 02:38:59.495148 4008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:59.482147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60323 (local address 127.0.98.62:35927)
0927 02:38:59.482334 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:59.482338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:59.483244 (+ 906us) server_negotiation.cc:407] Connection header received
0927 02:38:59.484825 (+ 1581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:59.484828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:59.484871 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:59.484938 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:59.486453 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:59.486963 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:59.488473 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:59.488768 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:59.488854 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:59.491628 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:59.491631 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:59.491633 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:59.491656 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:59.493010 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:59.493542 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:59.493545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:59.493546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:59.493622 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:59.493911 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:59.493913 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:59.493915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:59.494080 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:59.494151 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:59.494851 (+ 700us) server_negotiation.cc:299] Negotiation successful
0927 02:38:59.494983 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 02:38:59.495888 4162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:59.482569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35927 (local address 127.0.98.1:60323)
0927 02:38:59.483074 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:59.483134 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:59.484578 (+ 1444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:59.485152 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:59.485161 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:59.485616 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:38:59.486308 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:59.486330 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:59.487077 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:59.487080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:59.488336 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:59.488347 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:59.488922 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:59.488926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:59.489119 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:59.489743 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:38:59.489764 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:38:59.491426 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:38:59.493112 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:59.493117 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:59.493129 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:59.493425 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:59.493714 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:59.493716 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:59.493718 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:59.493841 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:59.494204 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:38:59.494211 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:38:59.494531 (+ 320us) client_negotiation.cc:769] Sending connection context
0927 02:38:59.494727 (+ 196us) client_negotiation.cc:240] Negotiation successful
0927 02:38:59.495016 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":130,"threads_started":1}
I20260927 02:38:59.497231 4160 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35927
I20260927 02:38:59.497467 4160 heartbeater.cc:464] Registering TS with master...
I20260927 02:38:59.498003 4160 heartbeater.cc:511] Master 127.0.98.62:35927 requested a full tablet report, sending...
I20260927 02:38:59.499552 3969 ts_manager.cc:194] Registered new tserver with Master: 38c11c8adb28462ab4a89c9a423b1ea4 (127.0.98.1:42539)
I20260927 02:38:59.500686 3969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60323
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:38:59.509480 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35927
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:38:59.616420 4165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:38:59.616742 4165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:38:59.616802 4165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:38:59.621672 4165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:38:59.621748 4165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:38:59.621834 4165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:38:59.627292 4165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:38:59.628335 4165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:38:59.629379 4165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:38:59.638774 4165 server_base.cc:1034] running on GCE node
W20260927 02:38:59.638662 4170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:38:59.638662 4171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:38:59.638669 4173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:38:59.639340 4165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:38:59.639953 4165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:38:59.641110 4165 hybrid_clock.cc:648] HybridClock initialized: now 1790476739641090 us; error 32 us; skew 500 ppm
Sep 27 02:38:59 dist-test-slave-zv8z krb5kdc[3924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:38:59.643741 4165 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:38:59.644564 4165 webserver.cc:460] Webserver started at http://127.0.98.2:33493/ using document root <none> and password file <none>
I20260927 02:38:59.645082 4165 fs_manager.cc:359] Metadata directory not provided
I20260927 02:38:59.645145 4165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:38:59.645332 4165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:38:59.647300 4165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/ts-1/data/instance:
uuid: "c9561dda0ccb4c62949575887b655643"
format_stamp: "Formatted at 2026-09-27 02:38:59 on dist-test-slave-zv8z"
server_key: "5cab4cbdfbc6ccec46f9ca32c096f27d"
server_key_iv: "bd8990f09e8bbf55083922636b36b4b7"
server_key_version: "encryptionkey@0"
I20260927 02:38:59.647805 4165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c9561dda0ccb4c62949575887b655643"
format_stamp: "Formatted at 2026-09-27 02:38:59 on dist-test-slave-zv8z"
server_key: "5cab4cbdfbc6ccec46f9ca32c096f27d"
server_key_iv: "bd8990f09e8bbf55083922636b36b4b7"
server_key_version: "encryptionkey@0"
I20260927 02:38:59.651269 4165 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:38:59.653542 4180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:59.661219 4165 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.003s sys 0.000s
I20260927 02:38:59.661343 4165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/ts-1/wal
uuid: "c9561dda0ccb4c62949575887b655643"
format_stamp: "Formatted at 2026-09-27 02:38:59 on dist-test-slave-zv8z"
server_key: "5cab4cbdfbc6ccec46f9ca32c096f27d"
server_key_iv: "bd8990f09e8bbf55083922636b36b4b7"
server_key_version: "encryptionkey@0"
I20260927 02:38:59.661466 4165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:38:59.688376 4165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:38:59.691548 4165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:38:59.691695 4165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:38:59.692210 4165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:38:59.692970 4165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:38:59.693020 4165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:59.693059 4165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:38:59.693084 4165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:38:59.701552 4165 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39859
I20260927 02:38:59.701583 4293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39859 every 8 connection(s)
I20260927 02:38:59.702433 4165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:38:59.705924 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 4165
I20260927 02:38:59.706089 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:38:59.706413 392 external_mini_cluster.cc:1428] Setting key 76816697d1ece6c66cd3e018eabcd857
Sep 27 02:38:59 dist-test-slave-zv8z krb5kdc[3924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:38:59.716248 4008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:59.704214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42717 (local address 127.0.98.62:35927)
0927 02:38:59.704360 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 02:38:59.704365 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:38:59.705267 (+ 902us) server_negotiation.cc:407] Connection header received
0927 02:38:59.706719 (+ 1452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:38:59.706723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:38:59.706779 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:38:59.706869 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:38:59.708214 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:59.708741 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:59.709890 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:38:59.710112 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:38:59.710199 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:59.712851 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:38:59.712853 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:38:59.712855 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:38:59.712879 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:38:59.714204 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:59.714765 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:59.714768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:59.714769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:59.714840 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:38:59.715230 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:38:59.715233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:38:59.715234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:38:59.715435 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:38:59.715519 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 02:38:59.716006 (+ 487us) server_negotiation.cc:299] Negotiation successful
0927 02:38:59.716130 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 02:38:59.717157 4296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:38:59.704649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35927 (local address 127.0.98.2:42717)
0927 02:38:59.705119 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0927 02:38:59.705165 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 02:38:59.706497 (+ 1332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:38:59.707074 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:38:59.707081 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:38:59.707547 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:38:59.708079 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:59.708094 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:59.708856 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:59.708860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:59.709778 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:38:59.709788 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:38:59.710245 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:38:59.710271 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:38:59.710443 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:38:59.710976 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:38:59.710992 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:38:59.712663 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:38:59.714333 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:59.714340 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:59.714356 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:59.714676 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:59.714930 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:38:59.714933 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:38:59.714935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:38:59.715114 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:38:59.715533 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:38:59.715538 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:38:59.715777 (+ 239us) client_negotiation.cc:769] Sending connection context
0927 02:38:59.715956 (+ 179us) client_negotiation.cc:240] Negotiation successful
0927 02:38:59.716306 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":116,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 02:38:59.718292 4294 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35927
I20260927 02:38:59.718510 4294 heartbeater.cc:464] Registering TS with master...
I20260927 02:38:59.718966 4294 heartbeater.cc:511] Master 127.0.98.62:35927 requested a full tablet report, sending...
I20260927 02:38:59.719988 3969 ts_manager.cc:194] Registered new tserver with Master: c9561dda0ccb4c62949575887b655643 (127.0.98.2:39859)
I20260927 02:38:59.720541 3969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42717
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab.
W20260927 02:39:00.213811 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.3: real 0.506s user 0.001s sys 0.002s
I20260927 02:39:00.214330 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35927
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:39:00.321460 4299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:39:00.321731 4299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:39:00.321790 4299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:39:00.325805 4299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:39:00.325862 4299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:39:00.325938 4299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:39:00.331075 4299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:39:00.332047 4299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:39:00.333039 4299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:39:00.342129 4305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:00.342082 4304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:00.342077 4307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:00.342285 4299 server_base.cc:1034] running on GCE node
I20260927 02:39:00.342818 4299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:39:00.343483 4299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:39:00.344655 4299 hybrid_clock.cc:648] HybridClock initialized: now 1790476740344630 us; error 34 us; skew 500 ppm
Sep 27 02:39:00 dist-test-slave-zv8z krb5kdc[3924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476740, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:39:00.347383 4299 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:39:00.348192 4299 webserver.cc:460] Webserver started at http://127.0.98.3:39835/ using document root <none> and password file <none>
I20260927 02:39:00.348706 4299 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:00.348770 4299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:00.348955 4299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:39:00.350987 4299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/ts-2/data/instance:
uuid: "e78e6332cb3f41239504d45add8b36bb"
format_stamp: "Formatted at 2026-09-27 02:39:00 on dist-test-slave-zv8z"
server_key: "cc682650b06f830f99f7bd29cc62b43c"
server_key_iv: "4f284d4e7dd65266d3fb6204d692d728"
server_key_version: "encryptionkey@0"
I20260927 02:39:00.351519 4299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e78e6332cb3f41239504d45add8b36bb"
format_stamp: "Formatted at 2026-09-27 02:39:00 on dist-test-slave-zv8z"
server_key: "cc682650b06f830f99f7bd29cc62b43c"
server_key_iv: "4f284d4e7dd65266d3fb6204d692d728"
server_key_version: "encryptionkey@0"
I20260927 02:39:00.355014 4299 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:39:00.357411 4314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:00.358745 4299 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:39:00.358851 4299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/ts-2/wal
uuid: "e78e6332cb3f41239504d45add8b36bb"
format_stamp: "Formatted at 2026-09-27 02:39:00 on dist-test-slave-zv8z"
server_key: "cc682650b06f830f99f7bd29cc62b43c"
server_key_iv: "4f284d4e7dd65266d3fb6204d692d728"
server_key_version: "encryptionkey@0"
I20260927 02:39:00.358944 4299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:00.368444 4299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:39:00.371310 4299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:39:00.371469 4299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:39:00.371981 4299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:39:00.372772 4299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:39:00.372820 4299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:00.372869 4299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:39:00.372897 4299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:00.381014 4299 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44879
I20260927 02:39:00.381033 4427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44879 every 8 connection(s)
I20260927 02:39:00.381944 4299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:39:00.390101 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 4299
I20260927 02:39:00.390223 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790476718023131-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:39:00.390560 392 external_mini_cluster.cc:1428] Setting key e6420c7a9a45a925b3dd9703e6489e16
Sep 27 02:39:00 dist-test-slave-zv8z krb5kdc[3924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476740, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:39:00.395846 4430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:00.383839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60169 (local address 127.0.98.62:35927)
0927 02:39:00.384093 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:00.384097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:00.384855 (+ 758us) server_negotiation.cc:407] Connection header received
0927 02:39:00.386267 (+ 1412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:00.386271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:00.386329 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:00.386417 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:00.387684 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:00.388263 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:00.389439 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:00.389677 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:00.389769 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:00.392571 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:00.392574 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:00.392576 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:00.392600 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:00.393896 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:00.394433 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:00.394436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:00.394437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:00.394523 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:00.394832 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:00.394834 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:00.394835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:00.395019 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:00.395094 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:00.395588 (+ 494us) server_negotiation.cc:299] Negotiation successful
0927 02:39:00.395700 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":79,"threads_started":1}
I20260927 02:39:00.396690 4431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:00.384227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35927 (local address 127.0.98.3:60169)
0927 02:39:00.384690 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:00.384734 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:00.386017 (+ 1283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:00.386624 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:00.386631 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:00.387014 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:00.387522 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:00.387537 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:00.388390 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:00.388393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:00.389310 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:00.389319 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:00.389833 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:00.389836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:00.389987 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:00.390726 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:00.390743 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:00.392415 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:00.394003 (+ 1588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:00.394008 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:00.394020 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:00.394307 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:00.394610 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:00.394613 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:00.394615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:00.394754 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:00.395100 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:00.395105 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:00.395364 (+ 259us) client_negotiation.cc:769] Sending connection context
0927 02:39:00.395531 (+ 167us) client_negotiation.cc:240] Negotiation successful
0927 02:39:00.395792 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":113,"threads_started":1}
I20260927 02:39:00.397891 4428 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35927
I20260927 02:39:00.398129 4428 heartbeater.cc:464] Registering TS with master...
I20260927 02:39:00.398660 4428 heartbeater.cc:511] Master 127.0.98.62:35927 requested a full tablet report, sending...
I20260927 02:39:00.399729 3969 ts_manager.cc:194] Registered new tserver with Master: e78e6332cb3f41239504d45add8b36bb (127.0.98.3:44879)
I20260927 02:39:00.400326 3969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60169
I20260927 02:39:00.404728 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 02:39:00 dist-test-slave-zv8z krb5kdc[3924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 02:39:00 dist-test-slave-zv8z krb5kdc[3924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:39:00.428735 4436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:00.419710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37572 (local address 127.0.98.1:42539)
0927 02:39:00.420020 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:00.420027 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:00.420050 (+ 23us) server_negotiation.cc:407] Connection header received
0927 02:39:00.420143 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:00.420149 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:00.420346 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:00.420469 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:00.421152 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:00.421967 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:00.422863 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:00.423241 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:00.423326 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:00.425776 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:00.425788 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:00.425805 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:00.425829 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:00.427336 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:00.427709 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:00.427713 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:00.427715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:00.427791 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:00.427986 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:00.427988 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:00.427989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:00.428210 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:00.428399 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:00.428486 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 02:39:00.428603 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":103,"threads_started":1}
Sep 27 02:39:00 dist-test-slave-zv8z krb5kdc[3924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:39:00.438784 4430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:00.431422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44488 (local address 127.0.98.62:35927)
0927 02:39:00.431563 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:00.431567 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:00.431627 (+ 60us) server_negotiation.cc:407] Connection header received
0927 02:39:00.431714 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:00.431716 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:00.431753 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:00.431830 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:00.432311 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:00.432815 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:00.433526 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:00.433729 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:00.433805 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:00.435924 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:00.435928 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:00.435931 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:00.435963 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:00.437094 (+ 1131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:00.437453 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:00.437455 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:00.437456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:00.437527 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:00.437857 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:00.437860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:00.437861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:00.438082 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:00.438215 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:00.438540 (+ 325us) server_negotiation.cc:299] Negotiation successful
0927 02:39:00.438655 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 02:39:00.439271 3969 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44488
I20260927 02:39:00.439841 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 4031
I20260927 02:39:00.444569 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 4165
I20260927 02:39:00.448877 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 4299
I20260927 02:39:00.453100 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 3937
2026-09-27T02:39:00Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2505 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:39:00 dist-test-slave-zv8z krb5kdc[4441](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:39:00 dist-test-slave-zv8z krb5kdc[4441](info): set up 2 sockets
Sep 27 02:39:00 dist-test-slave-zv8z krb5kdc[4441](info): commencing operation
krb5kdc: starting...
W20260927 02:39:01.402153 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/krb5kdc/test-user.keytab.
Sep 27 02:39:01 dist-test-slave-zv8z krb5kdc[4441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:39:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:39:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:39:01.516901 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44031
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:39:01.623111 4454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:39:01.623530 4454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:39:01.623608 4454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:39:01.628198 4454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:39:01.628261 4454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:39:01.628279 4454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:39:01.628295 4454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:39:01.628309 4454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:39:01.633628 4454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:39:01.634730 4454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:39:01.635738 4454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:39:01.643777 4462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:01.643823 4459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:01.643812 4460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:01.643991 4454 server_base.cc:1034] running on GCE node
I20260927 02:39:01.644621 4454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:39:01.645707 4454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:39:01.646875 4454 hybrid_clock.cc:648] HybridClock initialized: now 1790476741646859 us; error 34 us; skew 500 ppm
Sep 27 02:39:01 dist-test-slave-zv8z krb5kdc[4441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:39:01.649415 4454 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:39:01.650090 4454 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:01.650156 4454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:01.650372 4454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:39:01.652343 4454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/master-0/data/instance:
uuid: "600995d35b1c4be2b15268be72c1780e"
format_stamp: "Formatted at 2026-09-27 02:39:01 on dist-test-slave-zv8z"
server_key: "ffce9731669f7f9fdabe5a03e2510c69"
server_key_iv: "e825989ac73cb5f57ee0411b8d63c1eb"
server_key_version: "encryptionkey@0"
I20260927 02:39:01.652874 4454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/master-0/wal/instance:
uuid: "600995d35b1c4be2b15268be72c1780e"
format_stamp: "Formatted at 2026-09-27 02:39:01 on dist-test-slave-zv8z"
server_key: "ffce9731669f7f9fdabe5a03e2510c69"
server_key_iv: "e825989ac73cb5f57ee0411b8d63c1eb"
server_key_version: "encryptionkey@0"
I20260927 02:39:01.659030 4454 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260927 02:39:01.661669 4468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:01.662976 4454 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:39:01.663074 4454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "600995d35b1c4be2b15268be72c1780e"
format_stamp: "Formatted at 2026-09-27 02:39:01 on dist-test-slave-zv8z"
server_key: "ffce9731669f7f9fdabe5a03e2510c69"
server_key_iv: "e825989ac73cb5f57ee0411b8d63c1eb"
server_key_version: "encryptionkey@0"
I20260927 02:39:01.663160 4454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:01.673588 4454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:39:01.676738 4454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:39:01.676893 4454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:39:01.683605 4454 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44031
I20260927 02:39:01.683832 4520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44031 every 8 connection(s)
I20260927 02:39:01.684262 4454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:39:01.686990 4521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:39:01.692282 4521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 600995d35b1c4be2b15268be72c1780e: Bootstrap starting.
I20260927 02:39:01.692839 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 4454
I20260927 02:39:01.692971 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/master-0/wal/instance
I20260927 02:39:01.693228 392 external_mini_cluster.cc:1428] Setting key d5e4bd1b4cb555b5f0947029c87b2643
I20260927 02:39:01.695013 4521 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 600995d35b1c4be2b15268be72c1780e: Neither blocks nor log segments found. Creating new log.
I20260927 02:39:01.695868 4521 log.cc:824] T 00000000000000000000000000000000 P 600995d35b1c4be2b15268be72c1780e: Log is configured to *not* fsync() on all Append() calls
I20260927 02:39:01.697862 4521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 600995d35b1c4be2b15268be72c1780e: No bootstrap required, opened a new log
I20260927 02:39:01.701789 4521 raft_consensus.cc:348] T 00000000000000000000000000000000 P 600995d35b1c4be2b15268be72c1780e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "600995d35b1c4be2b15268be72c1780e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44031 } }
I20260927 02:39:01.702003 4521 raft_consensus.cc:374] T 00000000000000000000000000000000 P 600995d35b1c4be2b15268be72c1780e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:39:01.702039 4521 raft_consensus.cc:729] T 00000000000000000000000000000000 P 600995d35b1c4be2b15268be72c1780e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 600995d35b1c4be2b15268be72c1780e, State: Initialized, Role: FOLLOWER
Sep 27 02:39:01 dist-test-slave-zv8z krb5kdc[4441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:39:01.702513 4521 consensus_queue.cc:260] T 00000000000000000000000000000000 P 600995d35b1c4be2b15268be72c1780e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "600995d35b1c4be2b15268be72c1780e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44031 } }
I20260927 02:39:01.702636 4521 raft_consensus.cc:388] T 00000000000000000000000000000000 P 600995d35b1c4be2b15268be72c1780e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:39:01.702690 4521 raft_consensus.cc:482] T 00000000000000000000000000000000 P 600995d35b1c4be2b15268be72c1780e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:39:01.702782 4521 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 600995d35b1c4be2b15268be72c1780e [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:39:01.703821 4521 raft_consensus.cc:504] T 00000000000000000000000000000000 P 600995d35b1c4be2b15268be72c1780e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "600995d35b1c4be2b15268be72c1780e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44031 } }
I20260927 02:39:01.704180 4521 leader_election.cc:302] T 00000000000000000000000000000000 P 600995d35b1c4be2b15268be72c1780e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 600995d35b1c4be2b15268be72c1780e; no voters:
I20260927 02:39:01.704509 4521 leader_election.cc:288] T 00000000000000000000000000000000 P 600995d35b1c4be2b15268be72c1780e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:39:01.704597 4526 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 600995d35b1c4be2b15268be72c1780e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:39:01.704805 4526 raft_consensus.cc:686] T 00000000000000000000000000000000 P 600995d35b1c4be2b15268be72c1780e [term 1 LEADER]: Becoming Leader. State: Replica: 600995d35b1c4be2b15268be72c1780e, State: Running, Role: LEADER
I20260927 02:39:01.705145 4526 consensus_queue.cc:237] T 00000000000000000000000000000000 P 600995d35b1c4be2b15268be72c1780e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "600995d35b1c4be2b15268be72c1780e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44031 } }
I20260927 02:39:01.705814 4521 sys_catalog.cc:564] T 00000000000000000000000000000000 P 600995d35b1c4be2b15268be72c1780e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:39:01.706796 4528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 600995d35b1c4be2b15268be72c1780e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 600995d35b1c4be2b15268be72c1780e. Latest consensus state: current_term: 1 leader_uuid: "600995d35b1c4be2b15268be72c1780e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "600995d35b1c4be2b15268be72c1780e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44031 } } }
I20260927 02:39:01.706796 4527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 600995d35b1c4be2b15268be72c1780e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "600995d35b1c4be2b15268be72c1780e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "600995d35b1c4be2b15268be72c1780e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44031 } } }
I20260927 02:39:01.706931 4527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 600995d35b1c4be2b15268be72c1780e [sys.catalog]: This master's current role is: LEADER
I20260927 02:39:01.706931 4528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 600995d35b1c4be2b15268be72c1780e [sys.catalog]: This master's current role is: LEADER
I20260927 02:39:01.707366 4535 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:39:01.708757 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:01.694587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58040 (local address 127.0.98.62:44031)
0927 02:39:01.694995 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:01.695014 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:01.695057 (+ 43us) server_negotiation.cc:407] Connection header received
0927 02:39:01.695873 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:01.695898 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:01.696312 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:01.696617 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:01.697241 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:01.698652 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:01.699509 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:01.700207 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:01.700369 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:01.702877 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:01.702883 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:01.702899 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:01.702930 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:01.704909 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:01.705415 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:01.705421 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:01.705427 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:01.705561 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:01.705889 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:01.705892 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:01.705894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:01.706466 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:01.707369 (+ 903us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:01.707532 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 02:39:01.707844 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":108,"threads_started":1}
I20260927 02:39:01.711771 4535 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:39:01.717242 4535 catalog_manager.cc:1347] Generated new cluster ID: 1abafcb9ee9844f9ba5a41bed03b37e7
I20260927 02:39:01.717309 4535 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:39:01.742743 4535 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:39:01.743786 4535 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:39:01.751062 4535 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 600995d35b1c4be2b15268be72c1780e: Generated new TSK 0
I20260927 02:39:01.751766 4535 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:39:01.781775 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44031
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:39:01.906929 4547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:39:01.907173 4547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:39:01.907223 4547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:39:01.911237 4547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:39:01.911291 4547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:39:01.911351 4547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:39:01.916414 4547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:39:01.917368 4547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:39:01.918430 4547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:39:01.927580 4547 server_base.cc:1034] running on GCE node
W20260927 02:39:01.927410 4553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:01.927425 4552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:01.927418 4555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:01.928206 4547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:39:01.928809 4547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:39:01.929981 4547 hybrid_clock.cc:648] HybridClock initialized: now 1790476741929965 us; error 34 us; skew 500 ppm
Sep 27 02:39:01 dist-test-slave-zv8z krb5kdc[4441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:39:01.932852 4547 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:39:01.933549 4547 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:01.933617 4547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:01.933796 4547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:39:01.935880 4547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/ts-0/data/instance:
uuid: "5c8081e3bc104d6282bca18aa981dd35"
format_stamp: "Formatted at 2026-09-27 02:39:01 on dist-test-slave-zv8z"
server_key: "f4746a913bad0f8c1c56b1e7bb87fde8"
server_key_iv: "5d77be01c22db2a5918c0f705af10b60"
server_key_version: "encryptionkey@0"
I20260927 02:39:01.936405 4547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5c8081e3bc104d6282bca18aa981dd35"
format_stamp: "Formatted at 2026-09-27 02:39:01 on dist-test-slave-zv8z"
server_key: "f4746a913bad0f8c1c56b1e7bb87fde8"
server_key_iv: "5d77be01c22db2a5918c0f705af10b60"
server_key_version: "encryptionkey@0"
I20260927 02:39:01.940059 4547 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260927 02:39:01.942466 4561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:01.943755 4547 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 02:39:01.943866 4547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/ts-0/wal
uuid: "5c8081e3bc104d6282bca18aa981dd35"
format_stamp: "Formatted at 2026-09-27 02:39:01 on dist-test-slave-zv8z"
server_key: "f4746a913bad0f8c1c56b1e7bb87fde8"
server_key_iv: "5d77be01c22db2a5918c0f705af10b60"
server_key_version: "encryptionkey@0"
I20260927 02:39:01.943975 4547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:01.960777 4547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:39:01.964138 4547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:39:01.964337 4547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:39:01.964999 4547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:39:01.965842 4547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:39:01.965889 4547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:01.965965 4547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:39:01.966009 4547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:01.974471 4547 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45589
I20260927 02:39:01.974586 4674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45589 every 8 connection(s)
I20260927 02:39:01.975060 4547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:39:01.978549 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 4547
I20260927 02:39:01.978686 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:39:01.978972 392 external_mini_cluster.cc:1428] Setting key de5e40bb118725a6367c9bcd91add7c2
Sep 27 02:39:01 dist-test-slave-zv8z krb5kdc[4441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:39:01.989152 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:01.976744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36365 (local address 127.0.98.62:44031)
0927 02:39:01.976859 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:01.976863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:01.977720 (+ 857us) server_negotiation.cc:407] Connection header received
0927 02:39:01.979382 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:01.979386 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:01.979461 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:01.979547 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:01.981047 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:01.981573 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:01.982833 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:01.983177 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:01.983280 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:01.985602 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:01.985605 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:01.985608 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:01.985638 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:01.987200 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:01.987690 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:01.987694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:01.987696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:01.987780 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:01.988107 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:01.988110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:01.988112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:01.988324 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:01.988405 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:01.988929 (+ 524us) server_negotiation.cc:299] Negotiation successful
0927 02:39:01.989034 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 02:39:01.989996 4677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:01.977111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44031 (local address 127.0.98.1:36365)
0927 02:39:01.977568 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:01.977610 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:01.979142 (+ 1532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:01.979768 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:01.979776 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:01.980149 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:01.980866 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:01.980885 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:01.981718 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:01.981721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:01.982719 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:01.982728 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:01.983302 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:01.983305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:01.983438 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:01.984026 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:01.984041 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:01.985413 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:01.987315 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:01.987320 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:01.987333 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:01.987590 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:01.987868 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:01.987870 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:01.987871 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:01.988008 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:01.988408 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:01.988413 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:01.988692 (+ 279us) client_negotiation.cc:769] Sending connection context
0927 02:39:01.988858 (+ 166us) client_negotiation.cc:240] Negotiation successful
0927 02:39:01.989152 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":106,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 02:39:01.991272 4675 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44031
I20260927 02:39:01.991580 4675 heartbeater.cc:464] Registering TS with master...
I20260927 02:39:01.992110 4675 heartbeater.cc:511] Master 127.0.98.62:44031 requested a full tablet report, sending...
I20260927 02:39:01.993549 4485 ts_manager.cc:194] Registered new tserver with Master: 5c8081e3bc104d6282bca18aa981dd35 (127.0.98.1:45589)
I20260927 02:39:01.994716 4485 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36365
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:39:02.006044 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44031
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:39:02.117125 4680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:39:02.117404 4680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:39:02.117449 4680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:39:02.121521 4680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:39:02.121587 4680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:39:02.121650 4680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:39:02.127199 4680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:39:02.128314 4680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:39:02.129297 4680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:39:02.138690 4685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:02.138767 4686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:02.138729 4688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:02.138767 4680 server_base.cc:1034] running on GCE node
I20260927 02:39:02.139371 4680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:39:02.140041 4680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:39:02.141203 4680 hybrid_clock.cc:648] HybridClock initialized: now 1790476742141178 us; error 38 us; skew 500 ppm
Sep 27 02:39:02 dist-test-slave-zv8z krb5kdc[4441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:39:02.144105 4680 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:39:02.144970 4680 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:02.145044 4680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:02.145254 4680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:39:02.147481 4680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/ts-1/data/instance:
uuid: "0309f1e3abe14c9aa8531b7fb72bd9db"
format_stamp: "Formatted at 2026-09-27 02:39:02 on dist-test-slave-zv8z"
server_key: "0a63d97232193f83c52c974b4c52de59"
server_key_iv: "7550c9c9bfb3e7637ec168fffb8e4c9e"
server_key_version: "encryptionkey@0"
I20260927 02:39:02.148025 4680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0309f1e3abe14c9aa8531b7fb72bd9db"
format_stamp: "Formatted at 2026-09-27 02:39:02 on dist-test-slave-zv8z"
server_key: "0a63d97232193f83c52c974b4c52de59"
server_key_iv: "7550c9c9bfb3e7637ec168fffb8e4c9e"
server_key_version: "encryptionkey@0"
I20260927 02:39:02.151911 4680 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 02:39:02.154464 4694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:02.155956 4680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:39:02.156077 4680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/ts-1/wal
uuid: "0309f1e3abe14c9aa8531b7fb72bd9db"
format_stamp: "Formatted at 2026-09-27 02:39:02 on dist-test-slave-zv8z"
server_key: "0a63d97232193f83c52c974b4c52de59"
server_key_iv: "7550c9c9bfb3e7637ec168fffb8e4c9e"
server_key_version: "encryptionkey@0"
I20260927 02:39:02.156190 4680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:02.177799 4680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:39:02.181130 4680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:39:02.181406 4680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:39:02.182092 4680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:39:02.182999 4680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:39:02.183053 4680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:02.183108 4680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:39:02.183136 4680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:02.191725 4680 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36581
I20260927 02:39:02.191748 4807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36581 every 8 connection(s)
I20260927 02:39:02.192308 4680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:39:02.192572 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 4680
I20260927 02:39:02.192692 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:39:02.193009 392 external_mini_cluster.cc:1428] Setting key 2049f358183315a9ef06bd616678f473
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Sep 27 02:39:02 dist-test-slave-zv8z krb5kdc[4441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:39:02.209750 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:02.194001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58979 (local address 127.0.98.62:44031)
0927 02:39:02.194129 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:02.194133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:02.195241 (+ 1108us) server_negotiation.cc:407] Connection header received
0927 02:39:02.196816 (+ 1575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:02.196819 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:02.196872 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:02.196946 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:02.198162 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:02.199042 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:02.200193 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:02.200528 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:02.200640 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:02.205596 (+ 4956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:02.205599 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:02.205601 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:02.205630 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:02.207460 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:02.208100 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:02.208104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:02.208106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:02.208194 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:02.208595 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:02.208599 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:02.208600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:02.208830 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:02.208935 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:02.209449 (+ 514us) server_negotiation.cc:299] Negotiation successful
0927 02:39:02.209587 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 02:39:02.210659 4810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:02.194527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44031 (local address 127.0.98.2:58979)
0927 02:39:02.195058 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:02.195121 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:02.196596 (+ 1475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:02.197140 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:02.197147 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:02.197508 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:02.197988 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:02.198001 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:02.199178 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:02.199181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:02.200068 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:02.200077 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:02.200670 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:02.200672 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:02.200811 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:02.201511 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:02.201530 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:02.205384 (+ 3854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:02.207638 (+ 2254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:02.207644 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:02.207660 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:02.207970 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:02.208286 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:02.208289 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:02.208291 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:02.208495 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:02.208932 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:02.208939 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:02.209203 (+ 264us) client_negotiation.cc:769] Sending connection context
0927 02:39:02.209398 (+ 195us) client_negotiation.cc:240] Negotiation successful
0927 02:39:02.209750 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":114,"threads_started":1}
I20260927 02:39:02.212136 4808 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44031
I20260927 02:39:02.212404 4808 heartbeater.cc:464] Registering TS with master...
I20260927 02:39:02.212893 4808 heartbeater.cc:511] Master 127.0.98.62:44031 requested a full tablet report, sending...
I20260927 02:39:02.214046 4485 ts_manager.cc:194] Registered new tserver with Master: 0309f1e3abe14c9aa8531b7fb72bd9db (127.0.98.2:36581)
I20260927 02:39:02.214767 4485 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58979
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:39:02.221928 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44031
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:39:02.332032 4813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:39:02.332306 4813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:39:02.332350 4813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:39:02.336544 4813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:39:02.336609 4813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:39:02.336670 4813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:39:02.341899 4813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:39:02.342945 4813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:39:02.343901 4813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:39:02.353073 4819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:02.353092 4821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:02.353144 4813 server_base.cc:1034] running on GCE node
W20260927 02:39:02.353103 4818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:02.353660 4813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:39:02.354314 4813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:39:02.355489 4813 hybrid_clock.cc:648] HybridClock initialized: now 1790476742355461 us; error 39 us; skew 500 ppm
Sep 27 02:39:02 dist-test-slave-zv8z krb5kdc[4441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:39:02.358111 4813 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:39:02.358878 4813 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:02.358956 4813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:02.359174 4813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:39:02.361227 4813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/ts-2/data/instance:
uuid: "3fea486a82c24c64ad364e8b7ac2071b"
format_stamp: "Formatted at 2026-09-27 02:39:02 on dist-test-slave-zv8z"
server_key: "bf1c5162ad5779b6bc646a5f2ce04ed5"
server_key_iv: "5e3a652da89a90b914c653d400e6ad92"
server_key_version: "encryptionkey@0"
I20260927 02:39:02.361745 4813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3fea486a82c24c64ad364e8b7ac2071b"
format_stamp: "Formatted at 2026-09-27 02:39:02 on dist-test-slave-zv8z"
server_key: "bf1c5162ad5779b6bc646a5f2ce04ed5"
server_key_iv: "5e3a652da89a90b914c653d400e6ad92"
server_key_version: "encryptionkey@0"
I20260927 02:39:02.365345 4813 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 02:39:02.367786 4827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:02.368983 4813 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:39:02.369086 4813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/ts-2/wal
uuid: "3fea486a82c24c64ad364e8b7ac2071b"
format_stamp: "Formatted at 2026-09-27 02:39:02 on dist-test-slave-zv8z"
server_key: "bf1c5162ad5779b6bc646a5f2ce04ed5"
server_key_iv: "5e3a652da89a90b914c653d400e6ad92"
server_key_version: "encryptionkey@0"
I20260927 02:39:02.369179 4813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:02.400622 4813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:39:02.403751 4813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:39:02.403939 4813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:39:02.404517 4813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:39:02.405376 4813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:39:02.405433 4813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:02.405488 4813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:39:02.405512 4813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:02.413746 4813 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36107
I20260927 02:39:02.413774 4940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36107 every 8 connection(s)
I20260927 02:39:02.414336 4813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:39:02.418385 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 4813
I20260927 02:39:02.418524 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:39:02.418799 392 external_mini_cluster.cc:1428] Setting key 95367b48877d539c964e407506ca64ff
Sep 27 02:39:02 dist-test-slave-zv8z krb5kdc[4441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:39:02.427592 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:02.416048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35591 (local address 127.0.98.62:44031)
0927 02:39:02.416211 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:02.416216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:02.416976 (+ 760us) server_negotiation.cc:407] Connection header received
0927 02:39:02.418312 (+ 1336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:02.418316 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:02.418385 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:02.418508 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:02.419850 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:02.420385 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:02.421503 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:02.421716 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:02.421801 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:02.424154 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:02.424157 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:02.424159 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:02.424183 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:02.425623 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:02.426080 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:02.426083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:02.426086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:02.426160 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:02.426497 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:02.426500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:02.426502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:02.426726 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:02.426811 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:02.427313 (+ 502us) server_negotiation.cc:299] Negotiation successful
0927 02:39:02.427425 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 02:39:02.428349 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:02.416376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44031 (local address 127.0.98.3:35591)
0927 02:39:02.416816 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:02.416865 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:02.418081 (+ 1216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:02.418704 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:02.418711 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:02.419105 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:02.419684 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:02.419701 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:02.420509 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:02.420513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:02.421384 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:02.421392 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:02.421846 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:02.421849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:02.422037 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:02.422582 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:02.422598 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:02.423978 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:02.425729 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:02.425735 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:02.425750 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:02.425988 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:02.426238 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:02.426241 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:02.426243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:02.426412 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:02.426825 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:02.426832 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:02.427090 (+ 258us) client_negotiation.cc:769] Sending connection context
0927 02:39:02.427255 (+ 165us) client_negotiation.cc:240] Negotiation successful
0927 02:39:02.427559 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":110,"threads_started":1}
I20260927 02:39:02.429562 4941 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44031
I20260927 02:39:02.429772 4941 heartbeater.cc:464] Registering TS with master...
I20260927 02:39:02.430176 4941 heartbeater.cc:511] Master 127.0.98.62:44031 requested a full tablet report, sending...
I20260927 02:39:02.431206 4485 ts_manager.cc:194] Registered new tserver with Master: 3fea486a82c24c64ad364e8b7ac2071b (127.0.98.3:36107)
I20260927 02:39:02.431864 4485 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35591
I20260927 02:39:02.432600 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:39:02.441030 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:02.434671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58048 (local address 127.0.98.62:44031)
0927 02:39:02.434770 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:02.434773 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:02.434878 (+ 105us) server_negotiation.cc:407] Connection header received
0927 02:39:02.434971 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:02.434973 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:02.435007 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:02.435071 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:02.435674 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:02.436180 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:02.436975 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:02.437196 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:02.437281 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:02.438554 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:02.438557 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:02.438559 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:02.438580 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:02.439712 (+ 1132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:02.440072 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:02.440074 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:02.440075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:02.440138 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:02.440313 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:02.440315 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:02.440316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:02.440499 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:02.440643 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:02.440787 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 02:39:02.440872 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 02:39:02.443871 4485 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58048:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:39:02.446836 4485 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:39:02.461030 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:02.455027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36748 (local address 127.0.98.1:45589)
0927 02:39:02.456214 (+ 1187us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:02.456221 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:02.456245 (+ 24us) server_negotiation.cc:407] Connection header received
0927 02:39:02.456318 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:02.456323 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:02.456526 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:02.456682 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:39:02.456915 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:02.458031 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:02.459765 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:02.460370 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:02.460470 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:02.460722 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:02.460772 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 02:39:02.460861 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1066,"thread_start_us":89,"threads_started":1}
I20260927 02:39:02.461120 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:02.454865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45589 (local address 127.0.0.1:36748)
0927 02:39:02.455476 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:02.455924 (+ 448us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:02.455989 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:02.456671 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:02.456676 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:02.456695 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:39:02.456811 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:02.456816 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:02.458135 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:02.458138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:02.459652 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:02.459662 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:02.460479 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:02.460482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:02.460603 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:02.460642 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 02:39:02.460741 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 02:39:02.460861 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"mutex_wait_us":76,"thread_start_us":77,"threads_started":1}
I20260927 02:39:02.461344 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:02.454996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36107 (local address 127.0.0.1:35474)
0927 02:39:02.455688 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:02.455739 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:02.455860 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:02.457383 (+ 1523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:02.457386 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:02.457403 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:39:02.457479 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:02.457484 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:02.458712 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:02.458715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:02.459717 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:02.459728 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:02.460925 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:02.460929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:02.461051 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:02.461085 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 02:39:02.461152 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 02:39:02.461215 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"mutex_wait_us":46,"thread_start_us":103,"threads_started":1}
I20260927 02:39:02.461565 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:02.455204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36581 (local address 127.0.0.1:36828)
0927 02:39:02.455778 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:02.455792 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:02.455879 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:02.456902 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:02.456904 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:02.456914 (+ 10us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:39:02.456971 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:02.456974 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:02.458064 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:02.458067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:02.459704 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:02.459711 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:02.460925 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:02.460929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:02.461032 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:02.461085 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 02:39:02.461369 (+ 284us) client_negotiation.cc:240] Negotiation successful
0927 02:39:02.461439 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":206,"thread_start_us":94,"threads_started":1}
I20260927 02:39:02.461795 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:02.455603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35474 (local address 127.0.98.3:36107)
0927 02:39:02.456967 (+ 1364us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:02.456978 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:02.456999 (+ 21us) server_negotiation.cc:407] Connection header received
0927 02:39:02.457065 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:02.457070 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:02.457252 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:02.457396 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:39:02.457617 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:02.458574 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:02.459819 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:02.460632 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:02.460732 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:02.460865 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:02.461555 (+ 690us) server_negotiation.cc:299] Negotiation successful
0927 02:39:02.461654 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1265,"thread_start_us":144,"threads_started":1}
I20260927 02:39:02.462052 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:02.455615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36828 (local address 127.0.98.2:36581)
0927 02:39:02.456216 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:02.456226 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:02.456245 (+ 19us) server_negotiation.cc:407] Connection header received
0927 02:39:02.456317 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:02.456323 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:02.456530 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:02.456682 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:39:02.457073 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:02.457947 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:02.459820 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:02.460632 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:02.460731 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:02.460865 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:02.461812 (+ 947us) server_negotiation.cc:299] Negotiation successful
0927 02:39:02.461915 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":139,"threads_started":1}
I20260927 02:39:02.463982 4609 tablet_service.cc:1467] Processing CreateTablet for tablet 5ce189d6f7d94493803824053f3c126a (DEFAULT_TABLE table=test-table [id=5fbf24753c254dd7b3a27cfc94edda1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:39:02.464511 4742 tablet_service.cc:1467] Processing CreateTablet for tablet 5ce189d6f7d94493803824053f3c126a (DEFAULT_TABLE table=test-table [id=5fbf24753c254dd7b3a27cfc94edda1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:39:02.464936 4609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ce189d6f7d94493803824053f3c126a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:39:02.465375 4742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ce189d6f7d94493803824053f3c126a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:39:02.471037 4958 tablet_bootstrap.cc:492] T 5ce189d6f7d94493803824053f3c126a P 0309f1e3abe14c9aa8531b7fb72bd9db: Bootstrap starting.
I20260927 02:39:02.471673 4959 tablet_bootstrap.cc:492] T 5ce189d6f7d94493803824053f3c126a P 5c8081e3bc104d6282bca18aa981dd35: Bootstrap starting.
I20260927 02:39:02.473300 4958 tablet_bootstrap.cc:654] T 5ce189d6f7d94493803824053f3c126a P 0309f1e3abe14c9aa8531b7fb72bd9db: Neither blocks nor log segments found. Creating new log.
I20260927 02:39:02.472996 4875 tablet_service.cc:1467] Processing CreateTablet for tablet 5ce189d6f7d94493803824053f3c126a (DEFAULT_TABLE table=test-table [id=5fbf24753c254dd7b3a27cfc94edda1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:39:02.474009 4959 tablet_bootstrap.cc:654] T 5ce189d6f7d94493803824053f3c126a P 5c8081e3bc104d6282bca18aa981dd35: Neither blocks nor log segments found. Creating new log.
I20260927 02:39:02.474067 4875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ce189d6f7d94493803824053f3c126a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:39:02.474429 4958 log.cc:824] T 5ce189d6f7d94493803824053f3c126a P 0309f1e3abe14c9aa8531b7fb72bd9db: Log is configured to *not* fsync() on all Append() calls
I20260927 02:39:02.475232 4959 log.cc:824] T 5ce189d6f7d94493803824053f3c126a P 5c8081e3bc104d6282bca18aa981dd35: Log is configured to *not* fsync() on all Append() calls
I20260927 02:39:02.476460 4958 tablet_bootstrap.cc:492] T 5ce189d6f7d94493803824053f3c126a P 0309f1e3abe14c9aa8531b7fb72bd9db: No bootstrap required, opened a new log
I20260927 02:39:02.476701 4958 ts_tablet_manager.cc:1397] T 5ce189d6f7d94493803824053f3c126a P 0309f1e3abe14c9aa8531b7fb72bd9db: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 02:39:02.477126 4959 tablet_bootstrap.cc:492] T 5ce189d6f7d94493803824053f3c126a P 5c8081e3bc104d6282bca18aa981dd35: No bootstrap required, opened a new log
I20260927 02:39:02.477305 4959 ts_tablet_manager.cc:1397] T 5ce189d6f7d94493803824053f3c126a P 5c8081e3bc104d6282bca18aa981dd35: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 02:39:02.479542 4958 raft_consensus.cc:348] T 5ce189d6f7d94493803824053f3c126a P 0309f1e3abe14c9aa8531b7fb72bd9db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8081e3bc104d6282bca18aa981dd35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45589 } } peers { permanent_uuid: "3fea486a82c24c64ad364e8b7ac2071b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36107 } } peers { permanent_uuid: "0309f1e3abe14c9aa8531b7fb72bd9db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36581 } }
I20260927 02:39:02.479759 4958 raft_consensus.cc:374] T 5ce189d6f7d94493803824053f3c126a P 0309f1e3abe14c9aa8531b7fb72bd9db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:39:02.479827 4958 raft_consensus.cc:729] T 5ce189d6f7d94493803824053f3c126a P 0309f1e3abe14c9aa8531b7fb72bd9db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0309f1e3abe14c9aa8531b7fb72bd9db, State: Initialized, Role: FOLLOWER
I20260927 02:39:02.480448 4958 consensus_queue.cc:260] T 5ce189d6f7d94493803824053f3c126a P 0309f1e3abe14c9aa8531b7fb72bd9db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8081e3bc104d6282bca18aa981dd35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45589 } } peers { permanent_uuid: "3fea486a82c24c64ad364e8b7ac2071b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36107 } } peers { permanent_uuid: "0309f1e3abe14c9aa8531b7fb72bd9db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36581 } }
I20260927 02:39:02.481006 4959 raft_consensus.cc:348] T 5ce189d6f7d94493803824053f3c126a P 5c8081e3bc104d6282bca18aa981dd35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8081e3bc104d6282bca18aa981dd35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45589 } } peers { permanent_uuid: "3fea486a82c24c64ad364e8b7ac2071b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36107 } } peers { permanent_uuid: "0309f1e3abe14c9aa8531b7fb72bd9db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36581 } }
I20260927 02:39:02.481261 4959 raft_consensus.cc:374] T 5ce189d6f7d94493803824053f3c126a P 5c8081e3bc104d6282bca18aa981dd35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:39:02.481366 4959 raft_consensus.cc:729] T 5ce189d6f7d94493803824053f3c126a P 5c8081e3bc104d6282bca18aa981dd35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c8081e3bc104d6282bca18aa981dd35, State: Initialized, Role: FOLLOWER
I20260927 02:39:02.481601 4958 ts_tablet_manager.cc:1428] T 5ce189d6f7d94493803824053f3c126a P 0309f1e3abe14c9aa8531b7fb72bd9db: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 02:39:02.481673 4962 tablet_bootstrap.cc:492] T 5ce189d6f7d94493803824053f3c126a P 3fea486a82c24c64ad364e8b7ac2071b: Bootstrap starting.
I20260927 02:39:02.481889 4959 consensus_queue.cc:260] T 5ce189d6f7d94493803824053f3c126a P 5c8081e3bc104d6282bca18aa981dd35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8081e3bc104d6282bca18aa981dd35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45589 } } peers { permanent_uuid: "3fea486a82c24c64ad364e8b7ac2071b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36107 } } peers { permanent_uuid: "0309f1e3abe14c9aa8531b7fb72bd9db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36581 } }
I20260927 02:39:02.481974 4808 heartbeater.cc:503] Master 127.0.98.62:44031 was elected leader, sending a full tablet report...
I20260927 02:39:02.482590 4675 heartbeater.cc:503] Master 127.0.98.62:44031 was elected leader, sending a full tablet report...
I20260927 02:39:02.482973 4959 ts_tablet_manager.cc:1428] T 5ce189d6f7d94493803824053f3c126a P 5c8081e3bc104d6282bca18aa981dd35: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 02:39:02.484048 4962 tablet_bootstrap.cc:654] T 5ce189d6f7d94493803824053f3c126a P 3fea486a82c24c64ad364e8b7ac2071b: Neither blocks nor log segments found. Creating new log.
I20260927 02:39:02.485235 4962 log.cc:824] T 5ce189d6f7d94493803824053f3c126a P 3fea486a82c24c64ad364e8b7ac2071b: Log is configured to *not* fsync() on all Append() calls
I20260927 02:39:02.487110 4962 tablet_bootstrap.cc:492] T 5ce189d6f7d94493803824053f3c126a P 3fea486a82c24c64ad364e8b7ac2071b: No bootstrap required, opened a new log
I20260927 02:39:02.487264 4962 ts_tablet_manager.cc:1397] T 5ce189d6f7d94493803824053f3c126a P 3fea486a82c24c64ad364e8b7ac2071b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 02:39:02.490073 4962 raft_consensus.cc:348] T 5ce189d6f7d94493803824053f3c126a P 3fea486a82c24c64ad364e8b7ac2071b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8081e3bc104d6282bca18aa981dd35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45589 } } peers { permanent_uuid: "3fea486a82c24c64ad364e8b7ac2071b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36107 } } peers { permanent_uuid: "0309f1e3abe14c9aa8531b7fb72bd9db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36581 } }
I20260927 02:39:02.490337 4962 raft_consensus.cc:374] T 5ce189d6f7d94493803824053f3c126a P 3fea486a82c24c64ad364e8b7ac2071b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:39:02.490399 4962 raft_consensus.cc:729] T 5ce189d6f7d94493803824053f3c126a P 3fea486a82c24c64ad364e8b7ac2071b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fea486a82c24c64ad364e8b7ac2071b, State: Initialized, Role: FOLLOWER
I20260927 02:39:02.490813 4962 consensus_queue.cc:260] T 5ce189d6f7d94493803824053f3c126a P 3fea486a82c24c64ad364e8b7ac2071b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8081e3bc104d6282bca18aa981dd35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45589 } } peers { permanent_uuid: "3fea486a82c24c64ad364e8b7ac2071b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36107 } } peers { permanent_uuid: "0309f1e3abe14c9aa8531b7fb72bd9db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36581 } }
I20260927 02:39:02.491395 4941 heartbeater.cc:503] Master 127.0.98.62:44031 was elected leader, sending a full tablet report...
I20260927 02:39:02.491724 4962 ts_tablet_manager.cc:1428] T 5ce189d6f7d94493803824053f3c126a P 3fea486a82c24c64ad364e8b7ac2071b: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20260927 02:39:02.665179 4942 tablet.cc:2369] T 5ce189d6f7d94493803824053f3c126a P 3fea486a82c24c64ad364e8b7ac2071b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:39:02.693444 4809 tablet.cc:2369] T 5ce189d6f7d94493803824053f3c126a P 0309f1e3abe14c9aa8531b7fb72bd9db: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:39:02.726198 4676 tablet.cc:2369] T 5ce189d6f7d94493803824053f3c126a P 5c8081e3bc104d6282bca18aa981dd35: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:39:02.813814 4964 raft_consensus.cc:482] T 5ce189d6f7d94493803824053f3c126a P 5c8081e3bc104d6282bca18aa981dd35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:39:02.814029 4964 raft_consensus.cc:504] T 5ce189d6f7d94493803824053f3c126a P 5c8081e3bc104d6282bca18aa981dd35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8081e3bc104d6282bca18aa981dd35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45589 } } peers { permanent_uuid: "3fea486a82c24c64ad364e8b7ac2071b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36107 } } peers { permanent_uuid: "0309f1e3abe14c9aa8531b7fb72bd9db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36581 } }
I20260927 02:39:02.815203 4964 leader_election.cc:288] T 5ce189d6f7d94493803824053f3c126a P 5c8081e3bc104d6282bca18aa981dd35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fea486a82c24c64ad364e8b7ac2071b (127.0.98.3:36107), 0309f1e3abe14c9aa8531b7fb72bd9db (127.0.98.2:36581)
I20260927 02:39:02.819375 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:02.815574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56967 (local address 127.0.98.3:36107)
0927 02:39:02.815701 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:02.815705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:02.815886 (+ 181us) server_negotiation.cc:407] Connection header received
0927 02:39:02.816048 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:02.816050 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:02.816090 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:02.816152 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:39:02.816438 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:02.816954 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:02.818159 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:02.818822 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:02.818932 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:02.819002 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:02.819183 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 02:39:02.819247 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 02:39:02.819375 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:02.815607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36287 (local address 127.0.98.2:36581)
0927 02:39:02.815722 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:02.815726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:02.815899 (+ 173us) server_negotiation.cc:407] Connection header received
0927 02:39:02.816046 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:02.816048 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:02.816089 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:02.816152 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:39:02.816432 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:02.816931 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:02.818273 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:02.818861 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:02.818947 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:02.819002 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:02.819183 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 02:39:02.819247 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 02:39:02.819433 4967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:02.815481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36581 (local address 127.0.98.1:36287)
0927 02:39:02.815810 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:02.815821 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:02.815946 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:02.816176 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:02.816179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:02.816206 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:39:02.816325 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:02.816330 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:02.817056 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:02.817059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:02.818085 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:02.818097 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:02.818951 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:02.818954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:02.819083 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:02.819101 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 02:39:02.819162 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 02:39:02.819259 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":110,"spinlock_wait_cycles":9216,"thread_start_us":70,"threads_started":1}
I20260927 02:39:02.819437 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:02.815474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36107 (local address 127.0.98.1:56967)
0927 02:39:02.815797 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:02.815809 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:02.815944 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:02.816176 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:02.816179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:02.816203 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:39:02.816339 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:02.816344 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:02.817060 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:02.817063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:02.818063 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:02.818073 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:02.818932 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:02.818936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:02.819055 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:02.819083 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 02:39:02.819162 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 02:39:02.819260 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":106,"spinlock_wait_cycles":43776,"thread_start_us":111,"threads_started":1}
I20260927 02:39:02.820050 4762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ce189d6f7d94493803824053f3c126a" candidate_uuid: "5c8081e3bc104d6282bca18aa981dd35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0309f1e3abe14c9aa8531b7fb72bd9db" is_pre_election: true
I20260927 02:39:02.820106 4895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ce189d6f7d94493803824053f3c126a" candidate_uuid: "5c8081e3bc104d6282bca18aa981dd35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fea486a82c24c64ad364e8b7ac2071b" is_pre_election: true
I20260927 02:39:02.820325 4762 raft_consensus.cc:2454] T 5ce189d6f7d94493803824053f3c126a P 0309f1e3abe14c9aa8531b7fb72bd9db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c8081e3bc104d6282bca18aa981dd35 in term 0.
I20260927 02:39:02.820356 4895 raft_consensus.cc:2454] T 5ce189d6f7d94493803824053f3c126a P 3fea486a82c24c64ad364e8b7ac2071b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c8081e3bc104d6282bca18aa981dd35 in term 0.
I20260927 02:39:02.820770 4565 leader_election.cc:302] T 5ce189d6f7d94493803824053f3c126a P 5c8081e3bc104d6282bca18aa981dd35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fea486a82c24c64ad364e8b7ac2071b, 5c8081e3bc104d6282bca18aa981dd35; no voters:
I20260927 02:39:02.821012 4964 raft_consensus.cc:2781] T 5ce189d6f7d94493803824053f3c126a P 5c8081e3bc104d6282bca18aa981dd35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:39:02.821090 4964 raft_consensus.cc:482] T 5ce189d6f7d94493803824053f3c126a P 5c8081e3bc104d6282bca18aa981dd35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:39:02.821138 4964 raft_consensus.cc:3037] T 5ce189d6f7d94493803824053f3c126a P 5c8081e3bc104d6282bca18aa981dd35 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:39:02.822093 4964 raft_consensus.cc:504] T 5ce189d6f7d94493803824053f3c126a P 5c8081e3bc104d6282bca18aa981dd35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8081e3bc104d6282bca18aa981dd35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45589 } } peers { permanent_uuid: "3fea486a82c24c64ad364e8b7ac2071b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36107 } } peers { permanent_uuid: "0309f1e3abe14c9aa8531b7fb72bd9db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36581 } }
I20260927 02:39:02.822448 4964 leader_election.cc:288] T 5ce189d6f7d94493803824053f3c126a P 5c8081e3bc104d6282bca18aa981dd35 [CANDIDATE]: Term 1 election: Requested vote from peers 3fea486a82c24c64ad364e8b7ac2071b (127.0.98.3:36107), 0309f1e3abe14c9aa8531b7fb72bd9db (127.0.98.2:36581)
I20260927 02:39:02.822784 4762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ce189d6f7d94493803824053f3c126a" candidate_uuid: "5c8081e3bc104d6282bca18aa981dd35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0309f1e3abe14c9aa8531b7fb72bd9db"
I20260927 02:39:02.822778 4895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ce189d6f7d94493803824053f3c126a" candidate_uuid: "5c8081e3bc104d6282bca18aa981dd35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fea486a82c24c64ad364e8b7ac2071b"
I20260927 02:39:02.822890 4762 raft_consensus.cc:3037] T 5ce189d6f7d94493803824053f3c126a P 0309f1e3abe14c9aa8531b7fb72bd9db [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:39:02.822890 4895 raft_consensus.cc:3037] T 5ce189d6f7d94493803824053f3c126a P 3fea486a82c24c64ad364e8b7ac2071b [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:39:02.824076 4895 raft_consensus.cc:2454] T 5ce189d6f7d94493803824053f3c126a P 3fea486a82c24c64ad364e8b7ac2071b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c8081e3bc104d6282bca18aa981dd35 in term 1.
I20260927 02:39:02.824076 4762 raft_consensus.cc:2454] T 5ce189d6f7d94493803824053f3c126a P 0309f1e3abe14c9aa8531b7fb72bd9db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c8081e3bc104d6282bca18aa981dd35 in term 1.
I20260927 02:39:02.824348 4565 leader_election.cc:302] T 5ce189d6f7d94493803824053f3c126a P 5c8081e3bc104d6282bca18aa981dd35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fea486a82c24c64ad364e8b7ac2071b, 5c8081e3bc104d6282bca18aa981dd35; no voters:
I20260927 02:39:02.824507 4964 raft_consensus.cc:2781] T 5ce189d6f7d94493803824053f3c126a P 5c8081e3bc104d6282bca18aa981dd35 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:39:02.824708 4964 raft_consensus.cc:686] T 5ce189d6f7d94493803824053f3c126a P 5c8081e3bc104d6282bca18aa981dd35 [term 1 LEADER]: Becoming Leader. State: Replica: 5c8081e3bc104d6282bca18aa981dd35, State: Running, Role: LEADER
I20260927 02:39:02.825004 4964 consensus_queue.cc:237] T 5ce189d6f7d94493803824053f3c126a P 5c8081e3bc104d6282bca18aa981dd35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8081e3bc104d6282bca18aa981dd35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45589 } } peers { permanent_uuid: "3fea486a82c24c64ad364e8b7ac2071b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36107 } } peers { permanent_uuid: "0309f1e3abe14c9aa8531b7fb72bd9db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36581 } }
I20260927 02:39:02.828073 4485 catalog_manager.cc:5495] T 5ce189d6f7d94493803824053f3c126a P 5c8081e3bc104d6282bca18aa981dd35 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c8081e3bc104d6282bca18aa981dd35 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "5c8081e3bc104d6282bca18aa981dd35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c8081e3bc104d6282bca18aa981dd35" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fea486a82c24c64ad364e8b7ac2071b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0309f1e3abe14c9aa8531b7fb72bd9db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36581 } health_report { overall_health: UNKNOWN } } }
I20260927 02:39:02.914417 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:02.911079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36764 (local address 127.0.98.1:45589)
0927 02:39:02.911205 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:02.911209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:02.911223 (+ 14us) server_negotiation.cc:407] Connection header received
0927 02:39:02.911319 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:02.911322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:02.911375 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:02.911461 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:39:02.911653 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:02.912146 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:02.913003 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:02.913218 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:02.913289 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:02.913629 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:39:02.914024 (+ 395us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:39:02.914119 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:02.914218 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 02:39:02.914308 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 02:39:02.920122 4762 raft_consensus.cc:1261] T 5ce189d6f7d94493803824053f3c126a P 0309f1e3abe14c9aa8531b7fb72bd9db [term 1 FOLLOWER]: Refusing update from remote peer 5c8081e3bc104d6282bca18aa981dd35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:39:02.920125 4895 raft_consensus.cc:1261] T 5ce189d6f7d94493803824053f3c126a P 3fea486a82c24c64ad364e8b7ac2071b [term 1 FOLLOWER]: Refusing update from remote peer 5c8081e3bc104d6282bca18aa981dd35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:39:02.920760 4964 consensus_queue.cc:1035] T 5ce189d6f7d94493803824053f3c126a P 5c8081e3bc104d6282bca18aa981dd35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0309f1e3abe14c9aa8531b7fb72bd9db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:39:02.920945 4969 consensus_queue.cc:1035] T 5ce189d6f7d94493803824053f3c126a P 5c8081e3bc104d6282bca18aa981dd35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fea486a82c24c64ad364e8b7ac2071b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:39:02.929992 4973 mvcc.cc:205] Tried to move back new op lower bound from 7333792738994003968 to 7333792738613256192. Current Snapshot: MvccSnapshot[applied={T|T < 7333792738994003968}]
I20260927 02:39:02.938127 4484 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:39:02.938313 4484 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:39:02.940683 4484 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 600995d35b1c4be2b15268be72c1780e: Sending DeleteTablet for 3 replicas of tablet 5ce189d6f7d94493803824053f3c126a
I20260927 02:39:02.941407 4875 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ce189d6f7d94493803824053f3c126a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:39:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35474
I20260927 02:39:02.941579 4742 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ce189d6f7d94493803824053f3c126a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:39:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36828
I20260927 02:39:02.941589 4609 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ce189d6f7d94493803824053f3c126a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:39:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36748
I20260927 02:39:02.942003 4983 tablet_replica.cc:330] T 5ce189d6f7d94493803824053f3c126a P 5c8081e3bc104d6282bca18aa981dd35: stopping tablet replica
I20260927 02:39:02.942183 4982 tablet_replica.cc:330] T 5ce189d6f7d94493803824053f3c126a P 3fea486a82c24c64ad364e8b7ac2071b: stopping tablet replica
I20260927 02:39:02.942474 4983 raft_consensus.cc:2229] T 5ce189d6f7d94493803824053f3c126a P 5c8081e3bc104d6282bca18aa981dd35 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:39:02.942507 4982 raft_consensus.cc:2229] T 5ce189d6f7d94493803824053f3c126a P 3fea486a82c24c64ad364e8b7ac2071b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:39:02.942700 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 4547
I20260927 02:39:02.942828 4982 raft_consensus.cc:2258] T 5ce189d6f7d94493803824053f3c126a P 3fea486a82c24c64ad364e8b7ac2071b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:39:02.943658 4984 tablet_replica.cc:330] T 5ce189d6f7d94493803824053f3c126a P 0309f1e3abe14c9aa8531b7fb72bd9db: stopping tablet replica
I20260927 02:39:02.943914 4984 raft_consensus.cc:2229] T 5ce189d6f7d94493803824053f3c126a P 0309f1e3abe14c9aa8531b7fb72bd9db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:39:02.944168 4984 raft_consensus.cc:2258] T 5ce189d6f7d94493803824053f3c126a P 0309f1e3abe14c9aa8531b7fb72bd9db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:39:02.944254 4982 ts_tablet_manager.cc:1905] T 5ce189d6f7d94493803824053f3c126a P 3fea486a82c24c64ad364e8b7ac2071b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:39:02.945421 4984 ts_tablet_manager.cc:1905] T 5ce189d6f7d94493803824053f3c126a P 0309f1e3abe14c9aa8531b7fb72bd9db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:39:02.947152 4982 ts_tablet_manager.cc:1918] T 5ce189d6f7d94493803824053f3c126a P 3fea486a82c24c64ad364e8b7ac2071b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:39:02.947242 4982 log.cc:1196] T 5ce189d6f7d94493803824053f3c126a P 3fea486a82c24c64ad364e8b7ac2071b: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/ts-2/wal/wals/5ce189d6f7d94493803824053f3c126a
I20260927 02:39:02.947448 4984 ts_tablet_manager.cc:1918] T 5ce189d6f7d94493803824053f3c126a P 0309f1e3abe14c9aa8531b7fb72bd9db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:39:02.947544 4984 log.cc:1196] T 5ce189d6f7d94493803824053f3c126a P 0309f1e3abe14c9aa8531b7fb72bd9db: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790476718023131-392-0/minicluster-data/ts-1/wal/wals/5ce189d6f7d94493803824053f3c126a
I20260927 02:39:02.947541 4982 ts_tablet_manager.cc:1939] T 5ce189d6f7d94493803824053f3c126a P 3fea486a82c24c64ad364e8b7ac2071b: Deleting consensus metadata
I20260927 02:39:02.947793 4984 ts_tablet_manager.cc:1939] T 5ce189d6f7d94493803824053f3c126a P 0309f1e3abe14c9aa8531b7fb72bd9db: Deleting consensus metadata
I20260927 02:39:02.948688 4472 catalog_manager.cc:4841] TS 3fea486a82c24c64ad364e8b7ac2071b (127.0.98.3:36107): tablet 5ce189d6f7d94493803824053f3c126a (table test-table [id=5fbf24753c254dd7b3a27cfc94edda1d]) successfully deleted
I20260927 02:39:02.948686 4469 catalog_manager.cc:4841] TS 0309f1e3abe14c9aa8531b7fb72bd9db (127.0.98.2:36581): tablet 5ce189d6f7d94493803824053f3c126a (table test-table [id=5fbf24753c254dd7b3a27cfc94edda1d]) successfully deleted
W20260927 02:39:02.950024 4471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45589 (error 108)
I20260927 02:39:02.950068 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 4680
I20260927 02:39:02.951951 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:02.951700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45589 (local address 127.0.0.1:36774)
0927 02:39:02.951819 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:02.951880 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 02:39:02.952104 4471 catalog_manager.cc:4584] TS 5c8081e3bc104d6282bca18aa981dd35 (127.0.98.1:45589): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5ce189d6f7d94493803824053f3c126a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45589: connect: Connection refused (error 111)
I20260927 02:39:02.954988 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 4813
I20260927 02:39:02.959458 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 4454
2026-09-27T02:39:02Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2506 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T02:39:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:39:02Z Disabled control of system clock
I20260927 02:39:02.986670 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43815
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 02:39:03.092710 4989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:39:03.092958 4989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:39:03.092994 4989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:39:03.096964 4989 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:39:03.097030 4989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:39:03.097045 4989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:39:03.097060 4989 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:39:03.097074 4989 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:39:03.101749 4989 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43815
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:39:03.102843 4989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:39:03.104153 4989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:39:03.112313 4997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:03.112316 4994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:03.112316 4995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:03.112373 4989 server_base.cc:1034] running on GCE node
I20260927 02:39:03.113087 4989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:39:03.114125 4989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:39:03.115300 4989 hybrid_clock.cc:648] HybridClock initialized: now 1790476743115274 us; error 39 us; skew 500 ppm
I20260927 02:39:03.117141 4989 webserver.cc:460] Webserver started at http://127.0.98.62:37655/ using document root <none> and password file <none>
I20260927 02:39:03.117693 4989 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:03.117759 4989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:03.117939 4989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:39:03.120005 4989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/master-0/data/instance:
uuid: "614ebafb60b240009e9e93ba64ec6781"
format_stamp: "Formatted at 2026-09-27 02:39:03 on dist-test-slave-zv8z"
server_key: "1868926ca08f3ca7f1d6080daa0db821"
server_key_iv: "da0b7a6346a3336e9f383cc075264a50"
server_key_version: "encryptionkey@0"
I20260927 02:39:03.120561 4989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/master-0/wal/instance:
uuid: "614ebafb60b240009e9e93ba64ec6781"
format_stamp: "Formatted at 2026-09-27 02:39:03 on dist-test-slave-zv8z"
server_key: "1868926ca08f3ca7f1d6080daa0db821"
server_key_iv: "da0b7a6346a3336e9f383cc075264a50"
server_key_version: "encryptionkey@0"
I20260927 02:39:03.124276 4989 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:39:03.126679 5003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:03.127975 4989 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:39:03.128098 4989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "614ebafb60b240009e9e93ba64ec6781"
format_stamp: "Formatted at 2026-09-27 02:39:03 on dist-test-slave-zv8z"
server_key: "1868926ca08f3ca7f1d6080daa0db821"
server_key_iv: "da0b7a6346a3336e9f383cc075264a50"
server_key_version: "encryptionkey@0"
I20260927 02:39:03.128202 4989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:03.132639 4989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:39:03.133212 4989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:39:03.133361 4989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:39:03.139801 4989 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43815
I20260927 02:39:03.139810 5055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43815 every 8 connection(s)
I20260927 02:39:03.140676 4989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:39:03.141419 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 4989
I20260927 02:39:03.141526 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/master-0/wal/instance
I20260927 02:39:03.141875 392 external_mini_cluster.cc:1428] Setting key 3242b8468aa5168ddbfc22278027920b
I20260927 02:39:03.143903 5056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:39:03.147782 5059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:03.143222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43874 (local address 127.0.98.62:43815)
0927 02:39:03.144269 (+ 1047us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:03.144283 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:03.144319 (+ 36us) server_negotiation.cc:407] Connection header received
0927 02:39:03.144947 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:03.144963 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:03.145298 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:03.145579 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:03.146473 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:03.146478 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:03.146490 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:39:03.146501 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:03.146558 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:39:03.146627 (+ 69us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:03.146744 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:03.146838 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 02:39:03.147067 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":552,"threads_started":1}
I20260927 02:39:03.149829 5056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 614ebafb60b240009e9e93ba64ec6781: Bootstrap starting.
I20260927 02:39:03.151938 5056 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 614ebafb60b240009e9e93ba64ec6781: Neither blocks nor log segments found. Creating new log.
I20260927 02:39:03.152619 5056 log.cc:824] T 00000000000000000000000000000000 P 614ebafb60b240009e9e93ba64ec6781: Log is configured to *not* fsync() on all Append() calls
I20260927 02:39:03.154407 5056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 614ebafb60b240009e9e93ba64ec6781: No bootstrap required, opened a new log
I20260927 02:39:03.157819 5056 raft_consensus.cc:348] T 00000000000000000000000000000000 P 614ebafb60b240009e9e93ba64ec6781 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "614ebafb60b240009e9e93ba64ec6781" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43815 } }
I20260927 02:39:03.158030 5056 raft_consensus.cc:374] T 00000000000000000000000000000000 P 614ebafb60b240009e9e93ba64ec6781 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:39:03.158073 5056 raft_consensus.cc:729] T 00000000000000000000000000000000 P 614ebafb60b240009e9e93ba64ec6781 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 614ebafb60b240009e9e93ba64ec6781, State: Initialized, Role: FOLLOWER
I20260927 02:39:03.158528 5056 consensus_queue.cc:260] T 00000000000000000000000000000000 P 614ebafb60b240009e9e93ba64ec6781 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "614ebafb60b240009e9e93ba64ec6781" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43815 } }
I20260927 02:39:03.158651 5056 raft_consensus.cc:388] T 00000000000000000000000000000000 P 614ebafb60b240009e9e93ba64ec6781 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:39:03.158708 5056 raft_consensus.cc:482] T 00000000000000000000000000000000 P 614ebafb60b240009e9e93ba64ec6781 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:39:03.158799 5056 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 614ebafb60b240009e9e93ba64ec6781 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:39:03.159623 5056 raft_consensus.cc:504] T 00000000000000000000000000000000 P 614ebafb60b240009e9e93ba64ec6781 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "614ebafb60b240009e9e93ba64ec6781" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43815 } }
I20260927 02:39:03.159927 5056 leader_election.cc:302] T 00000000000000000000000000000000 P 614ebafb60b240009e9e93ba64ec6781 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 614ebafb60b240009e9e93ba64ec6781; no voters:
I20260927 02:39:03.160207 5056 leader_election.cc:288] T 00000000000000000000000000000000 P 614ebafb60b240009e9e93ba64ec6781 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:39:03.160318 5061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 614ebafb60b240009e9e93ba64ec6781 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:39:03.160528 5061 raft_consensus.cc:686] T 00000000000000000000000000000000 P 614ebafb60b240009e9e93ba64ec6781 [term 1 LEADER]: Becoming Leader. State: Replica: 614ebafb60b240009e9e93ba64ec6781, State: Running, Role: LEADER
I20260927 02:39:03.160898 5061 consensus_queue.cc:237] T 00000000000000000000000000000000 P 614ebafb60b240009e9e93ba64ec6781 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "614ebafb60b240009e9e93ba64ec6781" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43815 } }
I20260927 02:39:03.161387 5056 sys_catalog.cc:564] T 00000000000000000000000000000000 P 614ebafb60b240009e9e93ba64ec6781 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:39:03.162556 5063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 614ebafb60b240009e9e93ba64ec6781 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 614ebafb60b240009e9e93ba64ec6781. Latest consensus state: current_term: 1 leader_uuid: "614ebafb60b240009e9e93ba64ec6781" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "614ebafb60b240009e9e93ba64ec6781" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43815 } } }
I20260927 02:39:03.162621 5062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 614ebafb60b240009e9e93ba64ec6781 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "614ebafb60b240009e9e93ba64ec6781" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "614ebafb60b240009e9e93ba64ec6781" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43815 } } }
I20260927 02:39:03.162679 5063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 614ebafb60b240009e9e93ba64ec6781 [sys.catalog]: This master's current role is: LEADER
I20260927 02:39:03.162709 5062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 614ebafb60b240009e9e93ba64ec6781 [sys.catalog]: This master's current role is: LEADER
W20260927 02:39:03.165934 5076 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 614ebafb60b240009e9e93ba64ec6781: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:39:03.166007 5076 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:39:03.166065 5071 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:39:03.166766 5071 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:39:03.171828 5071 catalog_manager.cc:1347] Generated new cluster ID: bd980a14d5cc4deb80e38fe081609832
I20260927 02:39:03.171888 5071 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:39:03.189043 5071 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:39:03.189469 5071 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:39:03.198872 5071 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 614ebafb60b240009e9e93ba64ec6781: Generated new TSK 0
I20260927 02:39:03.199658 5071 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:39:03.202651 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43815
--builtin_ntp_servers=127.0.98.20:43555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 02:39:03.311484 5080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:39:03.311790 5080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:39:03.311867 5080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:39:03.315999 5080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:39:03.316056 5080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:39:03.316134 5080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:39:03.320899 5080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:39:03.321905 5080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:39:03.323300 5080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:39:03.332487 5086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:03.332504 5085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:03.332597 5080 server_base.cc:1034] running on GCE node
W20260927 02:39:03.332535 5088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:03.333130 5080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:39:03.333712 5080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:39:03.334867 5080 hybrid_clock.cc:648] HybridClock initialized: now 1790476743334838 us; error 41 us; skew 500 ppm
I20260927 02:39:03.336654 5080 webserver.cc:460] Webserver started at http://127.0.98.1:36731/ using document root <none> and password file <none>
I20260927 02:39:03.337196 5080 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:03.337255 5080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:03.337430 5080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:39:03.339448 5080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/ts-0/data/instance:
uuid: "ff11bdb467414f2f9f8d8056dad1a59f"
format_stamp: "Formatted at 2026-09-27 02:39:03 on dist-test-slave-zv8z"
server_key: "2addb37212c9f93cf559914588979067"
server_key_iv: "4d06fab1533cf97d75be9540aba162aa"
server_key_version: "encryptionkey@0"
I20260927 02:39:03.339960 5080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ff11bdb467414f2f9f8d8056dad1a59f"
format_stamp: "Formatted at 2026-09-27 02:39:03 on dist-test-slave-zv8z"
server_key: "2addb37212c9f93cf559914588979067"
server_key_iv: "4d06fab1533cf97d75be9540aba162aa"
server_key_version: "encryptionkey@0"
I20260927 02:39:03.343464 5080 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:39:03.345769 5094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:03.346987 5080 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:39:03.347105 5080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/ts-0/wal
uuid: "ff11bdb467414f2f9f8d8056dad1a59f"
format_stamp: "Formatted at 2026-09-27 02:39:03 on dist-test-slave-zv8z"
server_key: "2addb37212c9f93cf559914588979067"
server_key_iv: "4d06fab1533cf97d75be9540aba162aa"
server_key_version: "encryptionkey@0"
I20260927 02:39:03.347220 5080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:03.351540 5080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:39:03.352087 5080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:39:03.352238 5080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:39:03.352731 5080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:39:03.353555 5080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:39:03.353600 5080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:03.353647 5080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:39:03.353677 5080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:03.361871 5080 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35207
I20260927 02:39:03.361902 5207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35207 every 8 connection(s)
I20260927 02:39:03.362787 5080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:39:03.368422 5059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:03.364463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35345 (local address 127.0.98.62:43815)
0927 02:39:03.364614 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:03.364619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:03.365624 (+ 1005us) server_negotiation.cc:407] Connection header received
0927 02:39:03.366877 (+ 1253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:03.366880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:03.366916 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:03.367005 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:03.368061 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:03.368064 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:03.368066 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:39:03.368077 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:03.368104 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:39:03.368112 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:03.368186 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:03.368259 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 02:39:03.368293 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 02:39:03.368614 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 5080
I20260927 02:39:03.368743 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:39:03.369000 392 external_mini_cluster.cc:1428] Setting key 00f7995838e3d316df73bb6fa2bdba4d
I20260927 02:39:03.369279 5210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:03.364869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.1:35345)
0927 02:39:03.365490 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:03.365529 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:03.366676 (+ 1147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:03.367155 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:03.367161 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:03.367247 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:03.367805 (+ 558us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:39:03.367807 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:03.367846 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:39:03.367908 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:03.368172 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:03.368176 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:03.368180 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:39:03.368292 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 02:39:03.368492 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":104,"threads_started":1}
I20260927 02:39:03.370481 5208 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43815
I20260927 02:39:03.370692 5208 heartbeater.cc:464] Registering TS with master...
I20260927 02:39:03.370936 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43815
--builtin_ntp_servers=127.0.98.20:43555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 02:39:03.371048 5208 heartbeater.cc:511] Master 127.0.98.62:43815 requested a full tablet report, sending...
I20260927 02:39:03.372373 5019 ts_manager.cc:194] Registered new tserver with Master: ff11bdb467414f2f9f8d8056dad1a59f (127.0.98.1:35207)
W20260927 02:39:03.476230 5211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:39:03.476469 5211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:39:03.476506 5211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:39:03.480466 5211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:39:03.480516 5211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:39:03.480569 5211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:39:03.485105 5211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:39:03.486121 5211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:39:03.487604 5211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:39:03.497013 5219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:03.497056 5211 server_base.cc:1034] running on GCE node
W20260927 02:39:03.497001 5216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:03.496995 5217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:03.497546 5211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:39:03.498149 5211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:39:03.499322 5211 hybrid_clock.cc:648] HybridClock initialized: now 1790476743499299 us; error 34 us; skew 500 ppm
I20260927 02:39:03.501175 5211 webserver.cc:460] Webserver started at http://127.0.98.2:39431/ using document root <none> and password file <none>
I20260927 02:39:03.501714 5211 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:03.501773 5211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:03.501960 5211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:39:03.503935 5211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/ts-1/data/instance:
uuid: "55e43b635e80486c9b9be4bd7f2782a7"
format_stamp: "Formatted at 2026-09-27 02:39:03 on dist-test-slave-zv8z"
server_key: "51b5c266f601ba18c5e912806469470b"
server_key_iv: "d2ce535e6c0cf3872a54cac038338ab3"
server_key_version: "encryptionkey@0"
I20260927 02:39:03.504437 5211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/ts-1/wal/instance:
uuid: "55e43b635e80486c9b9be4bd7f2782a7"
format_stamp: "Formatted at 2026-09-27 02:39:03 on dist-test-slave-zv8z"
server_key: "51b5c266f601ba18c5e912806469470b"
server_key_iv: "d2ce535e6c0cf3872a54cac038338ab3"
server_key_version: "encryptionkey@0"
I20260927 02:39:03.507929 5211 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:39:03.510221 5225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:03.511463 5211 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:39:03.511564 5211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/ts-1/wal
uuid: "55e43b635e80486c9b9be4bd7f2782a7"
format_stamp: "Formatted at 2026-09-27 02:39:03 on dist-test-slave-zv8z"
server_key: "51b5c266f601ba18c5e912806469470b"
server_key_iv: "d2ce535e6c0cf3872a54cac038338ab3"
server_key_version: "encryptionkey@0"
I20260927 02:39:03.511696 5211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:03.516243 5211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:39:03.516798 5211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:39:03.516947 5211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:39:03.517467 5211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:39:03.518388 5211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:39:03.518435 5211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:03.518486 5211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:39:03.518522 5211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:03.526676 5211 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45533
I20260927 02:39:03.526726 5338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45533 every 8 connection(s)
I20260927 02:39:03.527525 5211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:39:03.533094 5059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:03.529278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45789 (local address 127.0.98.62:43815)
0927 02:39:03.529399 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:03.529402 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:03.530505 (+ 1103us) server_negotiation.cc:407] Connection header received
0927 02:39:03.531718 (+ 1213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:03.531721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:03.531757 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:03.531836 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:03.532789 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:03.532792 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:03.532794 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:39:03.532803 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:03.532820 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:39:03.532828 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:03.532894 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:03.532965 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 02:39:03.532996 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 02:39:03.533893 5341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:03.529698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.2:45789)
0927 02:39:03.530372 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:03.530411 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:03.531542 (+ 1131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:03.531959 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:03.531965 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:03.532050 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:03.532574 (+ 524us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:39:03.532575 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:03.532614 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:39:03.532680 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:03.532881 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:03.532885 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:03.532889 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:39:03.533000 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 02:39:03.533180 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":103,"threads_started":1}
I20260927 02:39:03.534930 5339 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43815
I20260927 02:39:03.535117 5339 heartbeater.cc:464] Registering TS with master...
I20260927 02:39:03.535487 5339 heartbeater.cc:511] Master 127.0.98.62:43815 requested a full tablet report, sending...
I20260927 02:39:03.536372 5019 ts_manager.cc:194] Registered new tserver with Master: 55e43b635e80486c9b9be4bd7f2782a7 (127.0.98.2:45533)
I20260927 02:39:03.536696 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 5211
I20260927 02:39:03.536816 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:39:03.537108 392 external_mini_cluster.cc:1428] Setting key 7b9fe84cdc2b9032efc338aa4e436d21
I20260927 02:39:03.539003 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43815
--builtin_ntp_servers=127.0.98.20:43555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 02:39:03.644203 5342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:39:03.644443 5342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:39:03.644481 5342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:39:03.648461 5342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:39:03.648510 5342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:39:03.648561 5342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:39:03.653148 5342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:39:03.654134 5342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:39:03.655745 5342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:39:03.666136 5347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:03.666124 5348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:03.666200 5342 server_base.cc:1034] running on GCE node
W20260927 02:39:03.666138 5350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:03.666719 5342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:39:03.667330 5342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:39:03.668475 5342 hybrid_clock.cc:648] HybridClock initialized: now 1790476743668433 us; error 53 us; skew 500 ppm
I20260927 02:39:03.670086 5342 webserver.cc:460] Webserver started at http://127.0.98.3:44125/ using document root <none> and password file <none>
I20260927 02:39:03.670634 5342 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:03.670698 5342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:03.670920 5342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:39:03.672820 5342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/ts-2/data/instance:
uuid: "26e5135f5b4f42518288f46472acbc1f"
format_stamp: "Formatted at 2026-09-27 02:39:03 on dist-test-slave-zv8z"
server_key: "7c4c96a93a46a693082aad456f981257"
server_key_iv: "1dd4959130e97fb567bf59944fe05487"
server_key_version: "encryptionkey@0"
I20260927 02:39:03.673316 5342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/ts-2/wal/instance:
uuid: "26e5135f5b4f42518288f46472acbc1f"
format_stamp: "Formatted at 2026-09-27 02:39:03 on dist-test-slave-zv8z"
server_key: "7c4c96a93a46a693082aad456f981257"
server_key_iv: "1dd4959130e97fb567bf59944fe05487"
server_key_version: "encryptionkey@0"
I20260927 02:39:03.676717 5342 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 02:39:03.679060 5356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:03.680181 5342 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:39:03.680280 5342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/ts-2/wal
uuid: "26e5135f5b4f42518288f46472acbc1f"
format_stamp: "Formatted at 2026-09-27 02:39:03 on dist-test-slave-zv8z"
server_key: "7c4c96a93a46a693082aad456f981257"
server_key_iv: "1dd4959130e97fb567bf59944fe05487"
server_key_version: "encryptionkey@0"
I20260927 02:39:03.680368 5342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:03.684621 5342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:39:03.685256 5342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:39:03.685413 5342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:39:03.685910 5342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:39:03.686748 5342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:39:03.686790 5342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:03.686826 5342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:39:03.686853 5342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:03.694650 5342 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39055
I20260927 02:39:03.694711 5469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39055 every 8 connection(s)
I20260927 02:39:03.695439 5342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:39:03.701103 5059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:03.697044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56457 (local address 127.0.98.62:43815)
0927 02:39:03.697208 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:03.697212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:03.698202 (+ 990us) server_negotiation.cc:407] Connection header received
0927 02:39:03.699586 (+ 1384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:03.699589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:03.699623 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:03.699692 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:03.700687 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:03.700689 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:03.700692 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:39:03.700701 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:03.700718 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:39:03.700726 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:03.700805 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:03.700937 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 02:39:03.700977 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 02:39:03.701884 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:03.697395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43815 (local address 127.0.98.3:56457)
0927 02:39:03.698045 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:03.698095 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:03.699401 (+ 1306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:03.699829 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:03.699835 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:03.699918 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:03.700444 (+ 526us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:39:03.700446 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:03.700484 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:39:03.700547 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:03.700810 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:03.700818 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:03.700824 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 02:39:03.700967 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 02:39:03.701190 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":113,"threads_started":1}
I20260927 02:39:03.702862 5470 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43815
I20260927 02:39:03.703042 5470 heartbeater.cc:464] Registering TS with master...
I20260927 02:39:03.703368 5470 heartbeater.cc:511] Master 127.0.98.62:43815 requested a full tablet report, sending...
I20260927 02:39:03.704281 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 5342
I20260927 02:39:03.704366 5019 ts_manager.cc:194] Registered new tserver with Master: 26e5135f5b4f42518288f46472acbc1f (127.0.98.3:39055)
I20260927 02:39:03.704399 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:39:03.704666 392 external_mini_cluster.cc:1428] Setting key 5666bc83106c8cb92200876f45b2387d
I20260927 02:39:03.706525 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:39:03.709424 5059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:03.708150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43878 (local address 127.0.98.62:43815)
0927 02:39:03.708268 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:03.708272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:03.708429 (+ 157us) server_negotiation.cc:407] Connection header received
0927 02:39:03.708521 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:03.708523 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:03.708547 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:03.708606 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:03.709141 (+ 535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:03.709144 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:03.709147 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:39:03.709158 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:03.709176 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:39:03.709184 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:03.709256 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:03.709310 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 02:39:03.709338 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 02:39:03.711861 5019 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43878:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:39:03.714704 5019 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:39:03.725040 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:03.722120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35207 (local address 127.0.0.1:48906)
0927 02:39:03.722544 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:03.722585 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:03.722671 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:03.723752 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:03.723756 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:03.723771 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:03.724456 (+ 685us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:39:03.724458 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:03.724466 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:39:03.724483 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:03.724783 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:03.724786 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:03.724790 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:39:03.724855 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 02:39:03.724911 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":62,"threads_started":1}
I20260927 02:39:03.725111 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:03.722288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48906 (local address 127.0.98.1:35207)
0927 02:39:03.722691 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:03.722702 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:03.722730 (+ 28us) server_negotiation.cc:407] Connection header received
0927 02:39:03.722783 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:03.722788 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:03.722976 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:03.723128 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:03.724597 (+ 1469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:03.724602 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:03.724610 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:39:03.724619 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:03.724662 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:39:03.724695 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:03.724798 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:03.724869 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 02:39:03.724942 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":101,"threads_started":1}
I20260927 02:39:03.725157 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:03.722205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39055 (local address 127.0.0.1:57624)
0927 02:39:03.722897 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:03.722910 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:03.722975 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:03.723567 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:03.723573 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:03.723649 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:03.724216 (+ 567us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:39:03.724218 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:03.724238 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:39:03.724303 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:03.724666 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:03.724674 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:03.724680 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 02:39:03.725002 (+ 322us) client_negotiation.cc:240] Negotiation successful
0927 02:39:03.725053 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":122,"threads_started":1}
I20260927 02:39:03.725522 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:03.722842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57624 (local address 127.0.98.3:39055)
0927 02:39:03.723167 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:03.723176 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:03.723198 (+ 22us) server_negotiation.cc:407] Connection header received
0927 02:39:03.723245 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:03.723249 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:03.723415 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:03.723918 (+ 503us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:03.724501 (+ 583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:03.724506 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:03.724517 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:39:03.724527 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:03.724562 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:39:03.724586 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:03.724683 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:03.725298 (+ 615us) server_negotiation.cc:299] Negotiation successful
0927 02:39:03.725380 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":128,"threads_started":1}
I20260927 02:39:03.725953 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:03.722811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45533 (local address 127.0.0.1:41360)
0927 02:39:03.723612 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:03.723624 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:03.723742 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:03.724616 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:03.724618 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:03.724633 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:03.725013 (+ 380us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:39:03.725014 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:03.725023 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:39:03.725046 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:03.725729 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:03.725732 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:03.725744 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 02:39:03.725814 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 02:39:03.725846 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"mutex_wait_us":576,"thread_start_us":89,"threads_started":1}
I20260927 02:39:03.726039 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:03.723024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41360 (local address 127.0.98.2:45533)
0927 02:39:03.723353 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:03.723363 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:03.723692 (+ 329us) server_negotiation.cc:407] Connection header received
0927 02:39:03.723853 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:03.723859 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:03.724014 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:03.724106 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:03.725513 (+ 1407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:03.725519 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:03.725528 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:39:03.725540 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:03.725574 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:39:03.725605 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:03.725692 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:03.725838 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 02:39:03.725897 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":135,"threads_started":1}
I20260927 02:39:03.727828 5142 tablet_service.cc:1467] Processing CreateTablet for tablet bc52974764be48d09d8c9b08f2f0fce9 (DEFAULT_TABLE table=test-table [id=e8ca32e95bab49f9a5d3dd41c14702c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:39:03.728065 5404 tablet_service.cc:1467] Processing CreateTablet for tablet bc52974764be48d09d8c9b08f2f0fce9 (DEFAULT_TABLE table=test-table [id=e8ca32e95bab49f9a5d3dd41c14702c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:39:03.728654 5142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc52974764be48d09d8c9b08f2f0fce9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:39:03.728379 5273 tablet_service.cc:1467] Processing CreateTablet for tablet bc52974764be48d09d8c9b08f2f0fce9 (DEFAULT_TABLE table=test-table [id=e8ca32e95bab49f9a5d3dd41c14702c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:39:03.729007 5404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc52974764be48d09d8c9b08f2f0fce9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:39:03.729230 5273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc52974764be48d09d8c9b08f2f0fce9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:39:03.734670 5487 tablet_bootstrap.cc:492] T bc52974764be48d09d8c9b08f2f0fce9 P ff11bdb467414f2f9f8d8056dad1a59f: Bootstrap starting.
I20260927 02:39:03.736095 5489 tablet_bootstrap.cc:492] T bc52974764be48d09d8c9b08f2f0fce9 P 55e43b635e80486c9b9be4bd7f2782a7: Bootstrap starting.
I20260927 02:39:03.736739 5487 tablet_bootstrap.cc:654] T bc52974764be48d09d8c9b08f2f0fce9 P ff11bdb467414f2f9f8d8056dad1a59f: Neither blocks nor log segments found. Creating new log.
I20260927 02:39:03.737404 5487 log.cc:824] T bc52974764be48d09d8c9b08f2f0fce9 P ff11bdb467414f2f9f8d8056dad1a59f: Log is configured to *not* fsync() on all Append() calls
I20260927 02:39:03.737751 5489 tablet_bootstrap.cc:654] T bc52974764be48d09d8c9b08f2f0fce9 P 55e43b635e80486c9b9be4bd7f2782a7: Neither blocks nor log segments found. Creating new log.
I20260927 02:39:03.738528 5489 log.cc:824] T bc52974764be48d09d8c9b08f2f0fce9 P 55e43b635e80486c9b9be4bd7f2782a7: Log is configured to *not* fsync() on all Append() calls
I20260927 02:39:03.739159 5488 tablet_bootstrap.cc:492] T bc52974764be48d09d8c9b08f2f0fce9 P 26e5135f5b4f42518288f46472acbc1f: Bootstrap starting.
I20260927 02:39:03.739269 5487 tablet_bootstrap.cc:492] T bc52974764be48d09d8c9b08f2f0fce9 P ff11bdb467414f2f9f8d8056dad1a59f: No bootstrap required, opened a new log
I20260927 02:39:03.739459 5487 ts_tablet_manager.cc:1397] T bc52974764be48d09d8c9b08f2f0fce9 P ff11bdb467414f2f9f8d8056dad1a59f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 02:39:03.740367 5489 tablet_bootstrap.cc:492] T bc52974764be48d09d8c9b08f2f0fce9 P 55e43b635e80486c9b9be4bd7f2782a7: No bootstrap required, opened a new log
I20260927 02:39:03.740545 5489 ts_tablet_manager.cc:1397] T bc52974764be48d09d8c9b08f2f0fce9 P 55e43b635e80486c9b9be4bd7f2782a7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 02:39:03.740860 5488 tablet_bootstrap.cc:654] T bc52974764be48d09d8c9b08f2f0fce9 P 26e5135f5b4f42518288f46472acbc1f: Neither blocks nor log segments found. Creating new log.
I20260927 02:39:03.741454 5488 log.cc:824] T bc52974764be48d09d8c9b08f2f0fce9 P 26e5135f5b4f42518288f46472acbc1f: Log is configured to *not* fsync() on all Append() calls
I20260927 02:39:03.742895 5488 tablet_bootstrap.cc:492] T bc52974764be48d09d8c9b08f2f0fce9 P 26e5135f5b4f42518288f46472acbc1f: No bootstrap required, opened a new log
I20260927 02:39:03.743046 5488 ts_tablet_manager.cc:1397] T bc52974764be48d09d8c9b08f2f0fce9 P 26e5135f5b4f42518288f46472acbc1f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 02:39:03.743876 5487 raft_consensus.cc:348] T bc52974764be48d09d8c9b08f2f0fce9 P ff11bdb467414f2f9f8d8056dad1a59f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff11bdb467414f2f9f8d8056dad1a59f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35207 } } peers { permanent_uuid: "26e5135f5b4f42518288f46472acbc1f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39055 } } peers { permanent_uuid: "55e43b635e80486c9b9be4bd7f2782a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45533 } }
I20260927 02:39:03.744144 5487 raft_consensus.cc:374] T bc52974764be48d09d8c9b08f2f0fce9 P ff11bdb467414f2f9f8d8056dad1a59f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:39:03.744191 5487 raft_consensus.cc:729] T bc52974764be48d09d8c9b08f2f0fce9 P ff11bdb467414f2f9f8d8056dad1a59f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff11bdb467414f2f9f8d8056dad1a59f, State: Initialized, Role: FOLLOWER
I20260927 02:39:03.744665 5487 consensus_queue.cc:260] T bc52974764be48d09d8c9b08f2f0fce9 P ff11bdb467414f2f9f8d8056dad1a59f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff11bdb467414f2f9f8d8056dad1a59f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35207 } } peers { permanent_uuid: "26e5135f5b4f42518288f46472acbc1f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39055 } } peers { permanent_uuid: "55e43b635e80486c9b9be4bd7f2782a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45533 } }
I20260927 02:39:03.744683 5489 raft_consensus.cc:348] T bc52974764be48d09d8c9b08f2f0fce9 P 55e43b635e80486c9b9be4bd7f2782a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff11bdb467414f2f9f8d8056dad1a59f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35207 } } peers { permanent_uuid: "26e5135f5b4f42518288f46472acbc1f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39055 } } peers { permanent_uuid: "55e43b635e80486c9b9be4bd7f2782a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45533 } }
I20260927 02:39:03.744903 5489 raft_consensus.cc:374] T bc52974764be48d09d8c9b08f2f0fce9 P 55e43b635e80486c9b9be4bd7f2782a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:39:03.744957 5489 raft_consensus.cc:729] T bc52974764be48d09d8c9b08f2f0fce9 P 55e43b635e80486c9b9be4bd7f2782a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55e43b635e80486c9b9be4bd7f2782a7, State: Initialized, Role: FOLLOWER
I20260927 02:39:03.745357 5208 heartbeater.cc:503] Master 127.0.98.62:43815 was elected leader, sending a full tablet report...
I20260927 02:39:03.745409 5489 consensus_queue.cc:260] T bc52974764be48d09d8c9b08f2f0fce9 P 55e43b635e80486c9b9be4bd7f2782a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff11bdb467414f2f9f8d8056dad1a59f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35207 } } peers { permanent_uuid: "26e5135f5b4f42518288f46472acbc1f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39055 } } peers { permanent_uuid: "55e43b635e80486c9b9be4bd7f2782a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45533 } }
I20260927 02:39:03.745668 5487 ts_tablet_manager.cc:1428] T bc52974764be48d09d8c9b08f2f0fce9 P ff11bdb467414f2f9f8d8056dad1a59f: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260927 02:39:03.746040 5339 heartbeater.cc:503] Master 127.0.98.62:43815 was elected leader, sending a full tablet report...
I20260927 02:39:03.746327 5489 ts_tablet_manager.cc:1428] T bc52974764be48d09d8c9b08f2f0fce9 P 55e43b635e80486c9b9be4bd7f2782a7: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 02:39:03.747586 5488 raft_consensus.cc:348] T bc52974764be48d09d8c9b08f2f0fce9 P 26e5135f5b4f42518288f46472acbc1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff11bdb467414f2f9f8d8056dad1a59f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35207 } } peers { permanent_uuid: "26e5135f5b4f42518288f46472acbc1f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39055 } } peers { permanent_uuid: "55e43b635e80486c9b9be4bd7f2782a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45533 } }
I20260927 02:39:03.747764 5488 raft_consensus.cc:374] T bc52974764be48d09d8c9b08f2f0fce9 P 26e5135f5b4f42518288f46472acbc1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:39:03.747812 5488 raft_consensus.cc:729] T bc52974764be48d09d8c9b08f2f0fce9 P 26e5135f5b4f42518288f46472acbc1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26e5135f5b4f42518288f46472acbc1f, State: Initialized, Role: FOLLOWER
I20260927 02:39:03.748148 5488 consensus_queue.cc:260] T bc52974764be48d09d8c9b08f2f0fce9 P 26e5135f5b4f42518288f46472acbc1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff11bdb467414f2f9f8d8056dad1a59f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35207 } } peers { permanent_uuid: "26e5135f5b4f42518288f46472acbc1f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39055 } } peers { permanent_uuid: "55e43b635e80486c9b9be4bd7f2782a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45533 } }
I20260927 02:39:03.748735 5470 heartbeater.cc:503] Master 127.0.98.62:43815 was elected leader, sending a full tablet report...
I20260927 02:39:03.749419 5488 ts_tablet_manager.cc:1428] T bc52974764be48d09d8c9b08f2f0fce9 P 26e5135f5b4f42518288f46472acbc1f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260927 02:39:03.778426 5340 tablet.cc:2369] T bc52974764be48d09d8c9b08f2f0fce9 P 55e43b635e80486c9b9be4bd7f2782a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:39:03.863971 5209 tablet.cc:2369] T bc52974764be48d09d8c9b08f2f0fce9 P ff11bdb467414f2f9f8d8056dad1a59f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:39:03.892977 5495 raft_consensus.cc:482] T bc52974764be48d09d8c9b08f2f0fce9 P 26e5135f5b4f42518288f46472acbc1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:39:03.893182 5495 raft_consensus.cc:504] T bc52974764be48d09d8c9b08f2f0fce9 P 26e5135f5b4f42518288f46472acbc1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff11bdb467414f2f9f8d8056dad1a59f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35207 } } peers { permanent_uuid: "26e5135f5b4f42518288f46472acbc1f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39055 } } peers { permanent_uuid: "55e43b635e80486c9b9be4bd7f2782a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45533 } }
I20260927 02:39:03.894402 5495 leader_election.cc:288] T bc52974764be48d09d8c9b08f2f0fce9 P 26e5135f5b4f42518288f46472acbc1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff11bdb467414f2f9f8d8056dad1a59f (127.0.98.1:35207), 55e43b635e80486c9b9be4bd7f2782a7 (127.0.98.2:45533)
I20260927 02:39:03.896267 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:03.894666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35207 (local address 127.0.98.3:34447)
0927 02:39:03.894799 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:03.894815 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:03.894906 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:03.895168 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:03.895171 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:03.895189 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:03.895768 (+ 579us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:39:03.895770 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:03.895779 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:39:03.895802 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:03.896032 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:03.896035 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:03.896038 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:39:03.896110 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 02:39:03.896141 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 02:39:03.896271 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:03.894805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34447 (local address 127.0.98.1:35207)
0927 02:39:03.894925 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:03.894929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:03.894947 (+ 18us) server_negotiation.cc:407] Connection header received
0927 02:39:03.894995 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:03.894998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:03.895035 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:03.895125 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:03.895921 (+ 796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:03.895925 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:03.895928 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:39:03.895940 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:03.895959 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:39:03.895967 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:03.896040 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:03.896104 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 02:39:03.896141 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 02:39:03.896519 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:03.894953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45533 (local address 127.0.98.3:34209)
0927 02:39:03.895301 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:03.895312 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:03.895398 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:03.895620 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:03.895623 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:03.895636 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:03.896012 (+ 376us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:39:03.896014 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:03.896023 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:39:03.896048 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:03.896267 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:03.896269 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:03.896272 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:39:03.896341 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 02:39:03.896377 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":115,"threads_started":1}
I20260927 02:39:03.896598 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:03.895077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34209 (local address 127.0.98.2:45533)
0927 02:39:03.895209 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:03.895213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:03.895393 (+ 180us) server_negotiation.cc:407] Connection header received
0927 02:39:03.895473 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:03.895476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:03.895514 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:03.895589 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:03.896154 (+ 565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:03.896158 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:03.896160 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:39:03.896171 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:03.896191 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:39:03.896199 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:03.896272 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:03.896456 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 02:39:03.896489 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 02:39:03.896934 5162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc52974764be48d09d8c9b08f2f0fce9" candidate_uuid: "26e5135f5b4f42518288f46472acbc1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff11bdb467414f2f9f8d8056dad1a59f" is_pre_election: true
I20260927 02:39:03.897208 5162 raft_consensus.cc:2454] T bc52974764be48d09d8c9b08f2f0fce9 P ff11bdb467414f2f9f8d8056dad1a59f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26e5135f5b4f42518288f46472acbc1f in term 0.
I20260927 02:39:03.897631 5357 leader_election.cc:302] T bc52974764be48d09d8c9b08f2f0fce9 P 26e5135f5b4f42518288f46472acbc1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26e5135f5b4f42518288f46472acbc1f, ff11bdb467414f2f9f8d8056dad1a59f; no voters:
I20260927 02:39:03.897907 5495 raft_consensus.cc:2781] T bc52974764be48d09d8c9b08f2f0fce9 P 26e5135f5b4f42518288f46472acbc1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:39:03.897986 5495 raft_consensus.cc:482] T bc52974764be48d09d8c9b08f2f0fce9 P 26e5135f5b4f42518288f46472acbc1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:39:03.898016 5495 raft_consensus.cc:3037] T bc52974764be48d09d8c9b08f2f0fce9 P 26e5135f5b4f42518288f46472acbc1f [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:39:03.897967 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc52974764be48d09d8c9b08f2f0fce9" candidate_uuid: "26e5135f5b4f42518288f46472acbc1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55e43b635e80486c9b9be4bd7f2782a7" is_pre_election: true
I20260927 02:39:03.898236 5293 raft_consensus.cc:2454] T bc52974764be48d09d8c9b08f2f0fce9 P 55e43b635e80486c9b9be4bd7f2782a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26e5135f5b4f42518288f46472acbc1f in term 0.
I20260927 02:39:03.899176 5495 raft_consensus.cc:504] T bc52974764be48d09d8c9b08f2f0fce9 P 26e5135f5b4f42518288f46472acbc1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff11bdb467414f2f9f8d8056dad1a59f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35207 } } peers { permanent_uuid: "26e5135f5b4f42518288f46472acbc1f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39055 } } peers { permanent_uuid: "55e43b635e80486c9b9be4bd7f2782a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45533 } }
I20260927 02:39:03.899478 5495 leader_election.cc:288] T bc52974764be48d09d8c9b08f2f0fce9 P 26e5135f5b4f42518288f46472acbc1f [CANDIDATE]: Term 1 election: Requested vote from peers ff11bdb467414f2f9f8d8056dad1a59f (127.0.98.1:35207), 55e43b635e80486c9b9be4bd7f2782a7 (127.0.98.2:45533)
I20260927 02:39:03.899772 5162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc52974764be48d09d8c9b08f2f0fce9" candidate_uuid: "26e5135f5b4f42518288f46472acbc1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff11bdb467414f2f9f8d8056dad1a59f"
I20260927 02:39:03.899887 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc52974764be48d09d8c9b08f2f0fce9" candidate_uuid: "26e5135f5b4f42518288f46472acbc1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55e43b635e80486c9b9be4bd7f2782a7"
I20260927 02:39:03.899886 5162 raft_consensus.cc:3037] T bc52974764be48d09d8c9b08f2f0fce9 P ff11bdb467414f2f9f8d8056dad1a59f [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:39:03.899997 5293 raft_consensus.cc:3037] T bc52974764be48d09d8c9b08f2f0fce9 P 55e43b635e80486c9b9be4bd7f2782a7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:39:03.900847 5162 raft_consensus.cc:2454] T bc52974764be48d09d8c9b08f2f0fce9 P ff11bdb467414f2f9f8d8056dad1a59f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26e5135f5b4f42518288f46472acbc1f in term 1.
I20260927 02:39:03.900964 5293 raft_consensus.cc:2454] T bc52974764be48d09d8c9b08f2f0fce9 P 55e43b635e80486c9b9be4bd7f2782a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26e5135f5b4f42518288f46472acbc1f in term 1.
I20260927 02:39:03.901203 5357 leader_election.cc:302] T bc52974764be48d09d8c9b08f2f0fce9 P 26e5135f5b4f42518288f46472acbc1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26e5135f5b4f42518288f46472acbc1f, ff11bdb467414f2f9f8d8056dad1a59f; no voters:
I20260927 02:39:03.901335 5495 raft_consensus.cc:2781] T bc52974764be48d09d8c9b08f2f0fce9 P 26e5135f5b4f42518288f46472acbc1f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:39:03.901530 5495 raft_consensus.cc:686] T bc52974764be48d09d8c9b08f2f0fce9 P 26e5135f5b4f42518288f46472acbc1f [term 1 LEADER]: Becoming Leader. State: Replica: 26e5135f5b4f42518288f46472acbc1f, State: Running, Role: LEADER
I20260927 02:39:03.901819 5495 consensus_queue.cc:237] T bc52974764be48d09d8c9b08f2f0fce9 P 26e5135f5b4f42518288f46472acbc1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff11bdb467414f2f9f8d8056dad1a59f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35207 } } peers { permanent_uuid: "26e5135f5b4f42518288f46472acbc1f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39055 } } peers { permanent_uuid: "55e43b635e80486c9b9be4bd7f2782a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45533 } }
I20260927 02:39:03.904747 5018 catalog_manager.cc:5495] T bc52974764be48d09d8c9b08f2f0fce9 P 26e5135f5b4f42518288f46472acbc1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 26e5135f5b4f42518288f46472acbc1f (127.0.98.3). New cstate: current_term: 1 leader_uuid: "26e5135f5b4f42518288f46472acbc1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff11bdb467414f2f9f8d8056dad1a59f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26e5135f5b4f42518288f46472acbc1f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55e43b635e80486c9b9be4bd7f2782a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45533 } health_report { overall_health: UNKNOWN } } }
I20260927 02:39:03.912614 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:03.911449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57640 (local address 127.0.98.3:39055)
0927 02:39:03.911557 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:03.911561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:03.911573 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:39:03.911671 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:03.911673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:03.911725 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:03.911789 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:03.912346 (+ 557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:03.912349 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:03.912351 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:39:03.912360 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:03.912376 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:39:03.912383 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:03.912455 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:03.912494 (+ 39us) server_negotiation.cc:299] Negotiation successful
0927 02:39:03.912524 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 02:39:03.918242 5162 raft_consensus.cc:1261] T bc52974764be48d09d8c9b08f2f0fce9 P ff11bdb467414f2f9f8d8056dad1a59f [term 1 FOLLOWER]: Refusing update from remote peer 26e5135f5b4f42518288f46472acbc1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:39:03.918242 5293 raft_consensus.cc:1261] T bc52974764be48d09d8c9b08f2f0fce9 P 55e43b635e80486c9b9be4bd7f2782a7 [term 1 FOLLOWER]: Refusing update from remote peer 26e5135f5b4f42518288f46472acbc1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:39:03.918999 5495 consensus_queue.cc:1035] T bc52974764be48d09d8c9b08f2f0fce9 P 26e5135f5b4f42518288f46472acbc1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff11bdb467414f2f9f8d8056dad1a59f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:39:03.919189 5497 consensus_queue.cc:1035] T bc52974764be48d09d8c9b08f2f0fce9 P 26e5135f5b4f42518288f46472acbc1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "55e43b635e80486c9b9be4bd7f2782a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:39:03.930897 5501 mvcc.cc:205] Tried to move back new op lower bound from 7333792743082139648 to 7333792743024283648. Current Snapshot: MvccSnapshot[applied={T|T < 7333792743082139648 or (T in {7333792743082139648})}]
I20260927 02:39:03.934892 5018 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:39:03.935027 5018 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:39:03.937119 5018 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 614ebafb60b240009e9e93ba64ec6781: Sending DeleteTablet for 3 replicas of tablet bc52974764be48d09d8c9b08f2f0fce9
I20260927 02:39:03.937842 5404 tablet_service.cc:1514] Processing DeleteTablet for tablet bc52974764be48d09d8c9b08f2f0fce9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:39:03 UTC) from {username='slave'} at 127.0.0.1:57624
I20260927 02:39:03.937882 5142 tablet_service.cc:1514] Processing DeleteTablet for tablet bc52974764be48d09d8c9b08f2f0fce9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:39:03 UTC) from {username='slave'} at 127.0.0.1:48906
I20260927 02:39:03.938308 5509 tablet_replica.cc:330] T bc52974764be48d09d8c9b08f2f0fce9 P 26e5135f5b4f42518288f46472acbc1f: stopping tablet replica
I20260927 02:39:03.938294 5273 tablet_service.cc:1514] Processing DeleteTablet for tablet bc52974764be48d09d8c9b08f2f0fce9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:39:03 UTC) from {username='slave'} at 127.0.0.1:41360
I20260927 02:39:03.938678 5509 raft_consensus.cc:2229] T bc52974764be48d09d8c9b08f2f0fce9 P 26e5135f5b4f42518288f46472acbc1f [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:39:03.938710 5511 tablet_replica.cc:330] T bc52974764be48d09d8c9b08f2f0fce9 P 55e43b635e80486c9b9be4bd7f2782a7: stopping tablet replica
I20260927 02:39:03.938932 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 5080
I20260927 02:39:03.940480 5511 raft_consensus.cc:2229] T bc52974764be48d09d8c9b08f2f0fce9 P 55e43b635e80486c9b9be4bd7f2782a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:39:03.940773 5511 raft_consensus.cc:2258] T bc52974764be48d09d8c9b08f2f0fce9 P 55e43b635e80486c9b9be4bd7f2782a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:39:03.941170 5509 raft_consensus.cc:2258] T bc52974764be48d09d8c9b08f2f0fce9 P 26e5135f5b4f42518288f46472acbc1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:39:03.942225 5511 ts_tablet_manager.cc:1905] T bc52974764be48d09d8c9b08f2f0fce9 P 55e43b635e80486c9b9be4bd7f2782a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:39:03.942394 5509 ts_tablet_manager.cc:1905] T bc52974764be48d09d8c9b08f2f0fce9 P 26e5135f5b4f42518288f46472acbc1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:39:03.944794 5511 ts_tablet_manager.cc:1918] T bc52974764be48d09d8c9b08f2f0fce9 P 55e43b635e80486c9b9be4bd7f2782a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:39:03.944806 5509 ts_tablet_manager.cc:1918] T bc52974764be48d09d8c9b08f2f0fce9 P 26e5135f5b4f42518288f46472acbc1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260927 02:39:03.945077 5004 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35207 (error 108)
I20260927 02:39:03.944893 5511 log.cc:1196] T bc52974764be48d09d8c9b08f2f0fce9 P 55e43b635e80486c9b9be4bd7f2782a7: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/ts-1/wal/wals/bc52974764be48d09d8c9b08f2f0fce9
I20260927 02:39:03.945258 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 5211
I20260927 02:39:03.945813 5509 log.cc:1196] T bc52974764be48d09d8c9b08f2f0fce9 P 26e5135f5b4f42518288f46472acbc1f: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790476718023131-392-0/minicluster-data/ts-2/wal/wals/bc52974764be48d09d8c9b08f2f0fce9
I20260927 02:39:03.946099 5509 ts_tablet_manager.cc:1939] T bc52974764be48d09d8c9b08f2f0fce9 P 26e5135f5b4f42518288f46472acbc1f: Deleting consensus metadata
I20260927 02:39:03.947145 5005 catalog_manager.cc:4841] TS 26e5135f5b4f42518288f46472acbc1f (127.0.98.3:39055): tablet bc52974764be48d09d8c9b08f2f0fce9 (table test-table [id=e8ca32e95bab49f9a5d3dd41c14702c0]) successfully deleted
I20260927 02:39:03.947691 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:03.947433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35207 (local address 127.0.0.1:48922)
0927 02:39:03.947551 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:03.947616 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 02:39:03.947839 5004 catalog_manager.cc:4584] TS ff11bdb467414f2f9f8d8056dad1a59f (127.0.98.1:35207): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bc52974764be48d09d8c9b08f2f0fce9: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35207: connect: Connection refused (error 111)
I20260927 02:39:03.951623 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 5342
I20260927 02:39:03.952687 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:03.952361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45533 (local address 127.0.0.1:41362)
0927 02:39:03.952566 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:03.952616 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:45533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
I20260927 02:39:03.956383 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 4989
2026-09-27T02:39:03Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (996 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 02:39:03.976102 392 mini_oidc.cc:152] Starting JWKS server
I20260927 02:39:03.977622 392 webserver.cc:460] Webserver started at https://127.0.0.1:46629/ using document root <none> and password file <none>
I20260927 02:39:03.977774 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 02:39:03.978014 392 webserver.cc:460] Webserver started at http://127.0.0.1:34437/ using document root <none> and password file <none>
2026-09-27T02:39:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:39:03Z Disabled control of system clock
I20260927 02:39:03.985741 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36891
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36891
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46629/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476718023131-392-0/testchainca.cert with env {}
W20260927 02:39:04.090755 5518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:39:04.091004 5518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:39:04.091042 5518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:39:04.094970 5518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:39:04.095028 5518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:39:04.095041 5518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:39:04.095057 5518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:39:04.095069 5518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 02:39:04.095083 5518 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 02:39:04.095094 5518 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46629/jwks.json
I20260927 02:39:04.100339 5518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36891
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46629/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476718023131-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:39:04.101305 5518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:39:04.102241 5518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:39:04.110474 5523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:04.110489 5526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:04.110460 5524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:04.110512 5518 server_base.cc:1034] running on GCE node
I20260927 02:39:04.111225 5518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:39:04.112238 5518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:39:04.113404 5518 hybrid_clock.cc:648] HybridClock initialized: now 1790476744113379 us; error 39 us; skew 500 ppm
I20260927 02:39:04.118922 5518 webserver.cc:460] Webserver started at http://127.0.98.62:38613/ using document root <none> and password file <none>
I20260927 02:39:04.119495 5518 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:04.119557 5518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:04.119756 5518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:39:04.121747 5518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476718023131-392-0/minicluster-data/master-0/data/instance:
uuid: "2c824fa70f1b465f82bd1986994b5685"
format_stamp: "Formatted at 2026-09-27 02:39:04 on dist-test-slave-zv8z"
server_key: "40e26f4f3f550622beb8c28bac6c41c1"
server_key_iv: "8883aebee90656a52bbef4558be2e774"
server_key_version: "encryptionkey@0"
I20260927 02:39:04.122295 5518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476718023131-392-0/minicluster-data/master-0/wal/instance:
uuid: "2c824fa70f1b465f82bd1986994b5685"
format_stamp: "Formatted at 2026-09-27 02:39:04 on dist-test-slave-zv8z"
server_key: "40e26f4f3f550622beb8c28bac6c41c1"
server_key_iv: "8883aebee90656a52bbef4558be2e774"
server_key_version: "encryptionkey@0"
I20260927 02:39:04.125656 5518 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 02:39:04.128224 5532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:04.129388 5518 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 02:39:04.129492 5518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "2c824fa70f1b465f82bd1986994b5685"
format_stamp: "Formatted at 2026-09-27 02:39:04 on dist-test-slave-zv8z"
server_key: "40e26f4f3f550622beb8c28bac6c41c1"
server_key_iv: "8883aebee90656a52bbef4558be2e774"
server_key_version: "encryptionkey@0"
I20260927 02:39:04.129580 5518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:04.160120 5518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:39:04.160899 5518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:39:04.161079 5518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:39:04.167667 5518 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36891
I20260927 02:39:04.167722 5584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36891 every 8 connection(s)
I20260927 02:39:04.168603 5518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:39:04.170830 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 5518
I20260927 02:39:04.170953 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476718023131-392-0/minicluster-data/master-0/wal/instance
I20260927 02:39:04.171267 392 external_mini_cluster.cc:1428] Setting key 6ac84565157f2c089492e8a186466beb
I20260927 02:39:04.173080 5585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:39:04.179118 5585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c824fa70f1b465f82bd1986994b5685: Bootstrap starting.
I20260927 02:39:04.180335 5588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:04.172616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58486 (local address 127.0.98.62:36891)
0927 02:39:04.173248 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:04.173264 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:04.173302 (+ 38us) server_negotiation.cc:407] Connection header received
0927 02:39:04.173953 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:04.173970 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:04.174406 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:04.174684 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:04.175024 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:04.175974 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:04.177143 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:04.177634 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:04.177759 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:04.178658 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:04.178663 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:04.178676 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:39:04.179056 (+ 380us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:04.179112 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:39:04.179131 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:04.179240 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:04.179367 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 02:39:04.179625 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":108,"threads_started":1}
I20260927 02:39:04.181481 5585 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c824fa70f1b465f82bd1986994b5685: Neither blocks nor log segments found. Creating new log.
I20260927 02:39:04.182327 5585 log.cc:824] T 00000000000000000000000000000000 P 2c824fa70f1b465f82bd1986994b5685: Log is configured to *not* fsync() on all Append() calls
I20260927 02:39:04.183187 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:39:04.184386 5585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c824fa70f1b465f82bd1986994b5685: No bootstrap required, opened a new log
I20260927 02:39:04.187861 5585 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c824fa70f1b465f82bd1986994b5685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c824fa70f1b465f82bd1986994b5685" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36891 } }
I20260927 02:39:04.188129 5585 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c824fa70f1b465f82bd1986994b5685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:39:04.188184 5585 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c824fa70f1b465f82bd1986994b5685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c824fa70f1b465f82bd1986994b5685, State: Initialized, Role: FOLLOWER
I20260927 02:39:04.188623 5585 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c824fa70f1b465f82bd1986994b5685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c824fa70f1b465f82bd1986994b5685" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36891 } }
I20260927 02:39:04.188762 5585 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c824fa70f1b465f82bd1986994b5685 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:39:04.188825 5585 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c824fa70f1b465f82bd1986994b5685 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:39:04.188920 5585 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c824fa70f1b465f82bd1986994b5685 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:39:04.189770 5585 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c824fa70f1b465f82bd1986994b5685 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c824fa70f1b465f82bd1986994b5685" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36891 } }
I20260927 02:39:04.190083 5585 leader_election.cc:302] T 00000000000000000000000000000000 P 2c824fa70f1b465f82bd1986994b5685 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c824fa70f1b465f82bd1986994b5685; no voters:
I20260927 02:39:04.190424 5585 leader_election.cc:288] T 00000000000000000000000000000000 P 2c824fa70f1b465f82bd1986994b5685 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:39:04.190518 5590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c824fa70f1b465f82bd1986994b5685 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:39:04.190723 5590 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c824fa70f1b465f82bd1986994b5685 [term 1 LEADER]: Becoming Leader. State: Replica: 2c824fa70f1b465f82bd1986994b5685, State: Running, Role: LEADER
I20260927 02:39:04.191006 5590 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c824fa70f1b465f82bd1986994b5685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c824fa70f1b465f82bd1986994b5685" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36891 } }
I20260927 02:39:04.191556 5585 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c824fa70f1b465f82bd1986994b5685 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:39:04.192581 5591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c824fa70f1b465f82bd1986994b5685 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c824fa70f1b465f82bd1986994b5685" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c824fa70f1b465f82bd1986994b5685" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36891 } } }
I20260927 02:39:04.192569 5592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c824fa70f1b465f82bd1986994b5685 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c824fa70f1b465f82bd1986994b5685. Latest consensus state: current_term: 1 leader_uuid: "2c824fa70f1b465f82bd1986994b5685" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c824fa70f1b465f82bd1986994b5685" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36891 } } }
I20260927 02:39:04.192708 5591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c824fa70f1b465f82bd1986994b5685 [sys.catalog]: This master's current role is: LEADER
I20260927 02:39:04.192708 5592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c824fa70f1b465f82bd1986994b5685 [sys.catalog]: This master's current role is: LEADER
I20260927 02:39:04.193120 5599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:39:04.196301 5599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:39:04.201987 5599 catalog_manager.cc:1347] Generated new cluster ID: 537af94214ba47d4a8dd7e0ea732e25b
I20260927 02:39:04.202071 5599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:39:04.226781 5599 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:39:04.228039 5599 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:39:04.238866 5599 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c824fa70f1b465f82bd1986994b5685: Generated new TSK 0
I20260927 02:39:04.239681 5599 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 02:39:04.279520 5615 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 02:39:04.279633 5615 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 02:39:04.280102 5588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:04.256082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58500 (local address 127.0.98.62:36891)
0927 02:39:04.256214 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:04.256219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:04.256472 (+ 253us) server_negotiation.cc:407] Connection header received
0927 02:39:04.256685 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:04.256688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:04.256735 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:04.256832 (+ 97us) server_negotiation.cc:226] Negotiated authn=JWT
0927 02:39:04.257100 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:04.257593 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:04.258867 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:04.259094 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:04.259186 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:04.259529 (+ 343us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 02:39:04.279665 (+ 20136us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46629/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 02:39:04.279992 (+ 327us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58500: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46629/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":49}
W20260927 02:39:04.280157 5588 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58500: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46629/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 02:39:04.281525 5614 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:04.255986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36891 (local address 127.0.0.1:58500)
0927 02:39:04.256357 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:04.256373 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:04.256505 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:04.256861 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:04.256865 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:04.256873 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0927 02:39:04.256999 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:04.257006 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:04.257703 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:04.257706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:04.258730 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:04.258741 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:04.259191 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:04.259210 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:04.259304 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:04.259354 (+ 50us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 02:39:04.279954 (+ 20600us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46629/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 02:39:04.280188 (+ 234us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:36891: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46629/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":107,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:36891: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46629/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 02:39:04.282281 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 5518
2026-09-27T02:39:04Z chronyd exiting
I20260927 02:39:04.318792 392 test_util.cc:175] -----------------------------------------------
I20260927 02:39:04.318884 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790476718023131-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (344 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 02:39:04.319918 392 mini_oidc.cc:152] Starting JWKS server
I20260927 02:39:04.320739 392 webserver.cc:460] Webserver started at https://127.0.0.1:45381/ using document root <none> and password file <none>
I20260927 02:39:04.320827 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 02:39:04.321038 392 webserver.cc:460] Webserver started at http://127.0.0.1:35559/ using document root <none> and password file <none>
2026-09-27T02:39:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:39:04Z Disabled control of system clock
I20260927 02:39:04.329028 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34895
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34895
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45381/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476718023131-392-0/testchainca.cert with env {}
W20260927 02:39:04.433956 5621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:39:04.434209 5621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:39:04.434271 5621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:39:04.438169 5621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:39:04.438227 5621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:39:04.438241 5621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:39:04.438277 5621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:39:04.438292 5621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 02:39:04.438311 5621 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 02:39:04.438337 5621 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45381/jwks.json
I20260927 02:39:04.443672 5621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34895
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45381/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476718023131-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:39:04.444973 5621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:39:04.445966 5621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:39:04.453830 5626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:04.453835 5627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:04.454047 5629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:04.454036 5621 server_base.cc:1034] running on GCE node
I20260927 02:39:04.454598 5621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:39:04.455587 5621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:39:04.456768 5621 hybrid_clock.cc:648] HybridClock initialized: now 1790476744456772 us; error 58 us; skew 500 ppm
I20260927 02:39:04.458547 5621 webserver.cc:460] Webserver started at http://127.0.98.62:34437/ using document root <none> and password file <none>
I20260927 02:39:04.459129 5621 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:04.459195 5621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:04.459368 5621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:39:04.461277 5621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476718023131-392-0/minicluster-data/master-0/data/instance:
uuid: "16d3d135f46f49ff88e109328c55b842"
format_stamp: "Formatted at 2026-09-27 02:39:04 on dist-test-slave-zv8z"
server_key: "1424aa6a48984863d32d468842e2bd54"
server_key_iv: "cac770f8f433c0255b896b75ae04a2e4"
server_key_version: "encryptionkey@0"
I20260927 02:39:04.461781 5621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476718023131-392-0/minicluster-data/master-0/wal/instance:
uuid: "16d3d135f46f49ff88e109328c55b842"
format_stamp: "Formatted at 2026-09-27 02:39:04 on dist-test-slave-zv8z"
server_key: "1424aa6a48984863d32d468842e2bd54"
server_key_iv: "cac770f8f433c0255b896b75ae04a2e4"
server_key_version: "encryptionkey@0"
I20260927 02:39:04.465190 5621 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 02:39:04.467429 5635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:04.468597 5621 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:39:04.468707 5621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "16d3d135f46f49ff88e109328c55b842"
format_stamp: "Formatted at 2026-09-27 02:39:04 on dist-test-slave-zv8z"
server_key: "1424aa6a48984863d32d468842e2bd54"
server_key_iv: "cac770f8f433c0255b896b75ae04a2e4"
server_key_version: "encryptionkey@0"
I20260927 02:39:04.468798 5621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:04.486521 5621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:39:04.487159 5621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:39:04.487319 5621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:39:04.493681 5621 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34895
I20260927 02:39:04.493705 5687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34895 every 8 connection(s)
I20260927 02:39:04.494580 5621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:39:04.497500 5688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:39:04.502766 5688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16d3d135f46f49ff88e109328c55b842: Bootstrap starting.
I20260927 02:39:04.504175 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 5621
I20260927 02:39:04.504304 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790476718023131-392-0/minicluster-data/master-0/wal/instance
I20260927 02:39:04.504580 392 external_mini_cluster.cc:1428] Setting key 3e0e804062b26249f9076ca268c8977e
I20260927 02:39:04.505381 5688 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16d3d135f46f49ff88e109328c55b842: Neither blocks nor log segments found. Creating new log.
I20260927 02:39:04.506177 5688 log.cc:824] T 00000000000000000000000000000000 P 16d3d135f46f49ff88e109328c55b842: Log is configured to *not* fsync() on all Append() calls
I20260927 02:39:04.508133 5688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16d3d135f46f49ff88e109328c55b842: No bootstrap required, opened a new log
I20260927 02:39:04.511651 5688 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16d3d135f46f49ff88e109328c55b842 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d3d135f46f49ff88e109328c55b842" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34895 } }
I20260927 02:39:04.511868 5688 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16d3d135f46f49ff88e109328c55b842 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:39:04.511931 5688 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16d3d135f46f49ff88e109328c55b842 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16d3d135f46f49ff88e109328c55b842, State: Initialized, Role: FOLLOWER
I20260927 02:39:04.512359 5688 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16d3d135f46f49ff88e109328c55b842 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d3d135f46f49ff88e109328c55b842" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34895 } }
I20260927 02:39:04.512501 5688 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16d3d135f46f49ff88e109328c55b842 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:39:04.512559 5688 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16d3d135f46f49ff88e109328c55b842 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:39:04.512650 5688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16d3d135f46f49ff88e109328c55b842 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:39:04.513368 5691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:04.505642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35726 (local address 127.0.98.62:34895)
0927 02:39:04.506306 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:04.506350 (+ 44us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:04.506385 (+ 35us) server_negotiation.cc:407] Connection header received
0927 02:39:04.507122 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:04.507141 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:04.507546 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:04.507857 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:04.508278 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:04.509254 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:04.510123 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:04.510633 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:04.510785 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:04.511597 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:04.511602 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:04.511614 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:39:04.512056 (+ 442us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:04.512111 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:39:04.512132 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:04.512213 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:04.512355 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 02:39:04.512598 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":130,"threads_started":1}
I20260927 02:39:04.513510 5688 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16d3d135f46f49ff88e109328c55b842 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d3d135f46f49ff88e109328c55b842" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34895 } }
I20260927 02:39:04.513825 5688 leader_election.cc:302] T 00000000000000000000000000000000 P 16d3d135f46f49ff88e109328c55b842 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16d3d135f46f49ff88e109328c55b842; no voters:
I20260927 02:39:04.514093 5688 leader_election.cc:288] T 00000000000000000000000000000000 P 16d3d135f46f49ff88e109328c55b842 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:39:04.514194 5693 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16d3d135f46f49ff88e109328c55b842 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:39:04.514386 5693 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16d3d135f46f49ff88e109328c55b842 [term 1 LEADER]: Becoming Leader. State: Replica: 16d3d135f46f49ff88e109328c55b842, State: Running, Role: LEADER
I20260927 02:39:04.514669 5693 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16d3d135f46f49ff88e109328c55b842 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d3d135f46f49ff88e109328c55b842" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34895 } }
I20260927 02:39:04.515468 5688 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16d3d135f46f49ff88e109328c55b842 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:39:04.516214 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:39:04.516530 5694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16d3d135f46f49ff88e109328c55b842 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16d3d135f46f49ff88e109328c55b842" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d3d135f46f49ff88e109328c55b842" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34895 } } }
I20260927 02:39:04.516649 5694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16d3d135f46f49ff88e109328c55b842 [sys.catalog]: This master's current role is: LEADER
I20260927 02:39:04.516949 5695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16d3d135f46f49ff88e109328c55b842 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16d3d135f46f49ff88e109328c55b842. Latest consensus state: current_term: 1 leader_uuid: "16d3d135f46f49ff88e109328c55b842" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d3d135f46f49ff88e109328c55b842" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34895 } } }
I20260927 02:39:04.517035 5695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16d3d135f46f49ff88e109328c55b842 [sys.catalog]: This master's current role is: LEADER
W20260927 02:39:04.520022 5708 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 16d3d135f46f49ff88e109328c55b842: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:39:04.520083 5708 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:39:04.520159 5702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:39:04.520885 5702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:39:04.525946 5702 catalog_manager.cc:1347] Generated new cluster ID: f56e40fd829c4f3c881f87c40782dfcd
I20260927 02:39:04.526010 5702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:39:04.547823 5702 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:39:04.548748 5702 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:39:04.557075 5702 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16d3d135f46f49ff88e109328c55b842: Generated new TSK 0
I20260927 02:39:04.557734 5702 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 02:39:04.597121 5718 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 02:39:04.597209 5718 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 02:39:04.597766 5691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:04.575022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35740 (local address 127.0.98.62:34895)
0927 02:39:04.575145 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:04.575150 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:04.575292 (+ 142us) server_negotiation.cc:407] Connection header received
0927 02:39:04.575478 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:04.575480 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:04.575523 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:04.575608 (+ 85us) server_negotiation.cc:226] Negotiated authn=JWT
0927 02:39:04.575825 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:04.576348 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:04.577558 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:04.577817 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:04.577897 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:04.578181 (+ 284us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 02:39:04.597440 (+ 19259us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45381/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 02:39:04.597657 (+ 217us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35740: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45381/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 02:39:04.597831 5691 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35740: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45381/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260927 02:39:04.597973 5717 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:04.574945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34895 (local address 127.0.0.1:35740)
0927 02:39:04.575208 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:04.575223 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:04.575370 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:04.575623 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:04.575626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:04.575634 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0927 02:39:04.575740 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:04.575746 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:04.576448 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:04.576452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:04.577442 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:04.577453 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:04.577907 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:04.577925 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:04.578013 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:04.578040 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 02:39:04.597629 (+ 19589us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45381/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 02:39:04.597804 (+ 175us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:34895: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45381/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":80,"threads_started":1}
I20260927 02:39:04.598591 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 5621
2026-09-27T02:39:04Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (317 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 02:39:04.636927 392 mini_oidc.cc:152] Starting JWKS server
I20260927 02:39:04.637755 392 webserver.cc:460] Webserver started at https://127.0.0.1:34647/ using document root <none> and password file <none>
I20260927 02:39:04.637853 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 02:39:04.638096 392 webserver.cc:460] Webserver started at http://127.0.0.1:33227/ using document root <none> and password file <none>
2026-09-27T02:39:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:39:04Z Disabled control of system clock
I20260927 02:39:04.646059 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45277
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45277
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34647/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 02:39:04.749115 5724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:39:04.749365 5724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:39:04.749403 5724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:39:04.753403 5724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:39:04.753463 5724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:39:04.753476 5724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:39:04.753491 5724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:39:04.753505 5724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 02:39:04.753520 5724 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 02:39:04.753543 5724 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34647/jwks.json
I20260927 02:39:04.758301 5724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45277
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34647/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:39:04.759342 5724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:39:04.760635 5724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:39:04.768380 5729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:04.768379 5732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:04.768394 5730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:04.768625 5724 server_base.cc:1034] running on GCE node
I20260927 02:39:04.769178 5724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:39:04.770227 5724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:39:04.771418 5724 hybrid_clock.cc:648] HybridClock initialized: now 1790476744771390 us; error 41 us; skew 500 ppm
I20260927 02:39:04.773121 5724 webserver.cc:460] Webserver started at http://127.0.98.62:33027/ using document root <none> and password file <none>
I20260927 02:39:04.773656 5724 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:04.773721 5724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:04.773901 5724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:39:04.775840 5724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790476718023131-392-0/minicluster-data/master-0/data/instance:
uuid: "c6d197e59eba40b7a9b94596b3607397"
format_stamp: "Formatted at 2026-09-27 02:39:04 on dist-test-slave-zv8z"
server_key: "d7f459aad5d1962d5afc8842ac324a72"
server_key_iv: "0370439a880009db44d4d515b1fc98cf"
server_key_version: "encryptionkey@0"
I20260927 02:39:04.776348 5724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790476718023131-392-0/minicluster-data/master-0/wal/instance:
uuid: "c6d197e59eba40b7a9b94596b3607397"
format_stamp: "Formatted at 2026-09-27 02:39:04 on dist-test-slave-zv8z"
server_key: "d7f459aad5d1962d5afc8842ac324a72"
server_key_iv: "0370439a880009db44d4d515b1fc98cf"
server_key_version: "encryptionkey@0"
I20260927 02:39:04.779698 5724 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 02:39:04.781832 5738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:04.782970 5724 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:39:04.783075 5724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "c6d197e59eba40b7a9b94596b3607397"
format_stamp: "Formatted at 2026-09-27 02:39:04 on dist-test-slave-zv8z"
server_key: "d7f459aad5d1962d5afc8842ac324a72"
server_key_iv: "0370439a880009db44d4d515b1fc98cf"
server_key_version: "encryptionkey@0"
I20260927 02:39:04.783160 5724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:04.803061 5724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:39:04.803700 5724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:39:04.803865 5724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:39:04.810191 5724 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45277
I20260927 02:39:04.810220 5790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45277 every 8 connection(s)
I20260927 02:39:04.811085 5724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:39:04.814131 5791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:39:04.819546 5791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6d197e59eba40b7a9b94596b3607397: Bootstrap starting.
I20260927 02:39:04.820907 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 5724
I20260927 02:39:04.821019 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790476718023131-392-0/minicluster-data/master-0/wal/instance
I20260927 02:39:04.821287 392 external_mini_cluster.cc:1428] Setting key fdde7380fffbbc0770d6a26886186058
I20260927 02:39:04.822216 5791 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6d197e59eba40b7a9b94596b3607397: Neither blocks nor log segments found. Creating new log.
I20260927 02:39:04.823055 5791 log.cc:824] T 00000000000000000000000000000000 P c6d197e59eba40b7a9b94596b3607397: Log is configured to *not* fsync() on all Append() calls
I20260927 02:39:04.824923 5791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6d197e59eba40b7a9b94596b3607397: No bootstrap required, opened a new log
I20260927 02:39:04.828167 5791 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6d197e59eba40b7a9b94596b3607397 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d197e59eba40b7a9b94596b3607397" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45277 } }
I20260927 02:39:04.828378 5791 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6d197e59eba40b7a9b94596b3607397 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:39:04.828426 5791 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6d197e59eba40b7a9b94596b3607397 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6d197e59eba40b7a9b94596b3607397, State: Initialized, Role: FOLLOWER
I20260927 02:39:04.828830 5791 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6d197e59eba40b7a9b94596b3607397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d197e59eba40b7a9b94596b3607397" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45277 } }
I20260927 02:39:04.828954 5791 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6d197e59eba40b7a9b94596b3607397 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:39:04.829015 5791 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6d197e59eba40b7a9b94596b3607397 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:39:04.829101 5791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6d197e59eba40b7a9b94596b3607397 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:39:04.829852 5794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:04.822477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54836 (local address 127.0.98.62:45277)
0927 02:39:04.823098 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:04.823111 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:04.823155 (+ 44us) server_negotiation.cc:407] Connection header received
0927 02:39:04.823940 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:04.823960 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:04.824310 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:04.824608 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:04.825019 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:04.825961 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:04.826841 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:04.827293 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:04.827445 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:04.828183 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:04.828188 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:04.828200 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:39:04.828617 (+ 417us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:04.828668 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:39:04.828688 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:04.828769 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:04.828913 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 02:39:04.829144 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":119,"threads_started":1}
I20260927 02:39:04.829897 5791 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6d197e59eba40b7a9b94596b3607397 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d197e59eba40b7a9b94596b3607397" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45277 } }
I20260927 02:39:04.830193 5791 leader_election.cc:302] T 00000000000000000000000000000000 P c6d197e59eba40b7a9b94596b3607397 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6d197e59eba40b7a9b94596b3607397; no voters:
I20260927 02:39:04.830474 5791 leader_election.cc:288] T 00000000000000000000000000000000 P c6d197e59eba40b7a9b94596b3607397 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:39:04.830559 5796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6d197e59eba40b7a9b94596b3607397 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:39:04.830711 5796 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6d197e59eba40b7a9b94596b3607397 [term 1 LEADER]: Becoming Leader. State: Replica: c6d197e59eba40b7a9b94596b3607397, State: Running, Role: LEADER
I20260927 02:39:04.830993 5796 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6d197e59eba40b7a9b94596b3607397 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d197e59eba40b7a9b94596b3607397" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45277 } }
I20260927 02:39:04.831662 5791 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6d197e59eba40b7a9b94596b3607397 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:39:04.832469 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:39:04.832501 5797 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6d197e59eba40b7a9b94596b3607397 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6d197e59eba40b7a9b94596b3607397. Latest consensus state: current_term: 1 leader_uuid: "c6d197e59eba40b7a9b94596b3607397" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d197e59eba40b7a9b94596b3607397" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45277 } } }
I20260927 02:39:04.832625 5797 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6d197e59eba40b7a9b94596b3607397 [sys.catalog]: This master's current role is: LEADER
I20260927 02:39:04.832930 5798 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6d197e59eba40b7a9b94596b3607397 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6d197e59eba40b7a9b94596b3607397" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d197e59eba40b7a9b94596b3607397" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45277 } } }
I20260927 02:39:04.833019 5798 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6d197e59eba40b7a9b94596b3607397 [sys.catalog]: This master's current role is: LEADER
I20260927 02:39:04.832979 5805 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:39:04.836071 5805 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:39:04.841470 5805 catalog_manager.cc:1347] Generated new cluster ID: 7034802c226a403c8926c07691447570
I20260927 02:39:04.841533 5805 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:39:04.855952 5805 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:39:04.856793 5805 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:39:04.861826 5805 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6d197e59eba40b7a9b94596b3607397: Generated new TSK 0
I20260927 02:39:04.862488 5805 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:39:04.879149 5794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:04.874341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54852 (local address 127.0.98.62:45277)
0927 02:39:04.874464 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:04.874468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:04.874621 (+ 153us) server_negotiation.cc:407] Connection header received
0927 02:39:04.874787 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:04.874789 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:04.874833 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:04.874949 (+ 116us) server_negotiation.cc:226] Negotiated authn=JWT
0927 02:39:04.875174 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:04.875682 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:04.876915 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:04.877128 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:04.877203 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:04.877492 (+ 289us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 02:39:04.878798 (+ 1306us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34647/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 02:39:04.879022 (+ 224us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54852: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34647/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":46}
W20260927 02:39:04.879210 5794 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54852: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34647/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 02:39:04.881019 5820 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:04.874233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45277 (local address 127.0.0.1:54852)
0927 02:39:04.874538 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:04.874554 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:04.874679 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:04.874966 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:04.874969 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:04.874977 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0927 02:39:04.875084 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:04.875090 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:04.875781 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:04.875784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:04.876785 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:04.876797 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:04.877220 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:04.877243 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:04.877331 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:04.877358 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 02:39:04.878958 (+ 1600us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34647/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 02:39:04.880854 (+ 1896us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:45277: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34647/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":1646,"thread_start_us":105,"threads_started":1}
I20260927 02:39:04.881660 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 5724
E20260927 02:39:04.890205 5821 webserver.cc:533] Webserver: sslize failed:
E20260927 02:39:04.890303 5821 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-27T02:39:04Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (282 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T02:39:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:39:04Z Disabled control of system clock
I20260927 02:39:04.927182 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39287
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39287
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476718023131-392-0/testchainca.cert with env {}
W20260927 02:39:05.031072 5825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:39:05.031327 5825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:39:05.031364 5825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:39:05.035390 5825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:39:05.035449 5825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:39:05.035463 5825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:39:05.035478 5825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:39:05.035492 5825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 02:39:05.035509 5825 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 02:39:05.035535 5825 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 02:39:05.040843 5825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39287
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476718023131-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:39:05.041841 5825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:39:05.042904 5825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:39:05.050909 5831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:05.050946 5830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:05.050947 5833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:05.051203 5825 server_base.cc:1034] running on GCE node
I20260927 02:39:05.051743 5825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:39:05.052798 5825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:39:05.053954 5825 hybrid_clock.cc:648] HybridClock initialized: now 1790476745053936 us; error 34 us; skew 500 ppm
I20260927 02:39:05.055748 5825 webserver.cc:460] Webserver started at http://127.0.98.62:37025/ using document root <none> and password file <none>
I20260927 02:39:05.056272 5825 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:05.056345 5825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:05.056535 5825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:39:05.058523 5825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476718023131-392-0/minicluster-data/master-0/data/instance:
uuid: "d6dbccaa8856473e9fe504b6af96f022"
format_stamp: "Formatted at 2026-09-27 02:39:05 on dist-test-slave-zv8z"
server_key: "ea9508138b04f5fb2d571117416bbf43"
server_key_iv: "d2b4792a2350bc8087fa147162463b8a"
server_key_version: "encryptionkey@0"
I20260927 02:39:05.059060 5825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476718023131-392-0/minicluster-data/master-0/wal/instance:
uuid: "d6dbccaa8856473e9fe504b6af96f022"
format_stamp: "Formatted at 2026-09-27 02:39:05 on dist-test-slave-zv8z"
server_key: "ea9508138b04f5fb2d571117416bbf43"
server_key_iv: "d2b4792a2350bc8087fa147162463b8a"
server_key_version: "encryptionkey@0"
I20260927 02:39:05.062569 5825 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260927 02:39:05.064827 5839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:05.065985 5825 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:39:05.066092 5825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "d6dbccaa8856473e9fe504b6af96f022"
format_stamp: "Formatted at 2026-09-27 02:39:05 on dist-test-slave-zv8z"
server_key: "ea9508138b04f5fb2d571117416bbf43"
server_key_iv: "d2b4792a2350bc8087fa147162463b8a"
server_key_version: "encryptionkey@0"
I20260927 02:39:05.066183 5825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:05.085513 5825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:39:05.086175 5825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:39:05.086387 5825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:39:05.092708 5825 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39287
I20260927 02:39:05.092736 5891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39287 every 8 connection(s)
I20260927 02:39:05.093604 5825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:39:05.096633 5892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:39:05.102181 5892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6dbccaa8856473e9fe504b6af96f022: Bootstrap starting.
I20260927 02:39:05.102329 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 5825
I20260927 02:39:05.102451 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790476718023131-392-0/minicluster-data/master-0/wal/instance
I20260927 02:39:05.102701 392 external_mini_cluster.cc:1428] Setting key c0bf2239a12edfd1077d3b3d6b419569
I20260927 02:39:05.104566 5892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6dbccaa8856473e9fe504b6af96f022: Neither blocks nor log segments found. Creating new log.
I20260927 02:39:05.105254 5892 log.cc:824] T 00000000000000000000000000000000 P d6dbccaa8856473e9fe504b6af96f022: Log is configured to *not* fsync() on all Append() calls
I20260927 02:39:05.107188 5892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6dbccaa8856473e9fe504b6af96f022: No bootstrap required, opened a new log
I20260927 02:39:05.111119 5892 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6dbccaa8856473e9fe504b6af96f022 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6dbccaa8856473e9fe504b6af96f022" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39287 } }
I20260927 02:39:05.111351 5892 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6dbccaa8856473e9fe504b6af96f022 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:39:05.111387 5892 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6dbccaa8856473e9fe504b6af96f022 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6dbccaa8856473e9fe504b6af96f022, State: Initialized, Role: FOLLOWER
I20260927 02:39:05.111651 5895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:05.103894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51858 (local address 127.0.98.62:39287)
0927 02:39:05.104484 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:05.104498 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:05.104531 (+ 33us) server_negotiation.cc:407] Connection header received
0927 02:39:05.105202 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:05.105220 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:05.105654 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:05.106010 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:05.106390 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:05.107359 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:05.108507 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:05.108989 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:05.109100 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:05.109909 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:05.109913 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:05.109925 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:39:05.110322 (+ 397us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:05.110375 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:39:05.110395 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:05.110465 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:05.110599 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 02:39:05.110856 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":112,"threads_started":1}
I20260927 02:39:05.111807 5892 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6dbccaa8856473e9fe504b6af96f022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6dbccaa8856473e9fe504b6af96f022" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39287 } }
I20260927 02:39:05.111950 5892 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6dbccaa8856473e9fe504b6af96f022 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:39:05.112015 5892 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6dbccaa8856473e9fe504b6af96f022 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:39:05.112107 5892 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6dbccaa8856473e9fe504b6af96f022 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:39:05.113006 5892 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6dbccaa8856473e9fe504b6af96f022 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6dbccaa8856473e9fe504b6af96f022" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39287 } }
I20260927 02:39:05.113349 5892 leader_election.cc:302] T 00000000000000000000000000000000 P d6dbccaa8856473e9fe504b6af96f022 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6dbccaa8856473e9fe504b6af96f022; no voters:
I20260927 02:39:05.113643 5892 leader_election.cc:288] T 00000000000000000000000000000000 P d6dbccaa8856473e9fe504b6af96f022 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:39:05.113731 5897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6dbccaa8856473e9fe504b6af96f022 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:39:05.113893 5897 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6dbccaa8856473e9fe504b6af96f022 [term 1 LEADER]: Becoming Leader. State: Replica: d6dbccaa8856473e9fe504b6af96f022, State: Running, Role: LEADER
I20260927 02:39:05.114187 5897 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6dbccaa8856473e9fe504b6af96f022 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6dbccaa8856473e9fe504b6af96f022" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39287 } }
I20260927 02:39:05.114487 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:39:05.114861 5892 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6dbccaa8856473e9fe504b6af96f022 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:39:05.115736 5898 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6dbccaa8856473e9fe504b6af96f022 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6dbccaa8856473e9fe504b6af96f022" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6dbccaa8856473e9fe504b6af96f022" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39287 } } }
I20260927 02:39:05.115861 5898 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6dbccaa8856473e9fe504b6af96f022 [sys.catalog]: This master's current role is: LEADER
I20260927 02:39:05.116163 5899 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6dbccaa8856473e9fe504b6af96f022 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6dbccaa8856473e9fe504b6af96f022. Latest consensus state: current_term: 1 leader_uuid: "d6dbccaa8856473e9fe504b6af96f022" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6dbccaa8856473e9fe504b6af96f022" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39287 } } }
I20260927 02:39:05.116262 5899 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6dbccaa8856473e9fe504b6af96f022 [sys.catalog]: This master's current role is: LEADER
I20260927 02:39:05.117061 5906 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:39:05.119576 5906 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:39:05.124640 5906 catalog_manager.cc:1347] Generated new cluster ID: 675a78a79dbd4d29ae0127c4dbe5294a
I20260927 02:39:05.124701 5906 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:39:05.153415 5906 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:39:05.154716 5906 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:39:05.164330 5906 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6dbccaa8856473e9fe504b6af96f022: Generated new TSK 0
I20260927 02:39:05.165181 5906 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:39:05.196254 5895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:05.173429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51866 (local address 127.0.98.62:39287)
0927 02:39:05.173551 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:05.173556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:05.173705 (+ 149us) server_negotiation.cc:407] Connection header received
0927 02:39:05.173865 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:05.173868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:05.173921 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:05.174033 (+ 112us) server_negotiation.cc:226] Negotiated authn=JWT
0927 02:39:05.174335 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:05.175160 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:05.176024 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:05.176372 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:05.176471 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:05.176784 (+ 313us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 02:39:05.195809 (+ 19025us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 02:39:05.196110 (+ 301us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51866: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":47}
W20260927 02:39:05.196344 5895 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51866: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 02:39:05.196362 5921 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:05.173328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39287 (local address 127.0.0.1:51866)
0927 02:39:05.173626 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:05.173639 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:05.173740 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:05.174086 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:05.174089 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:05.174097 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0927 02:39:05.174211 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:05.174217 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:05.175281 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:05.175284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:05.175919 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:05.175929 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:05.176509 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:05.176528 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:05.176600 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:05.176625 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 02:39:05.196039 (+ 19414us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 02:39:05.196211 (+ 172us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:39287: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":108,"threads_started":1}
I20260927 02:39:05.197072 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 5825
2026-09-27T02:39:05Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (294 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (118 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (128 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:39:05 dist-test-slave-zv8z krb5kdc[5933](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:39:05 dist-test-slave-zv8z krb5kdc[5933](info): set up 2 sockets
Sep 27 02:39:05 dist-test-slave-zv8z krb5kdc[5933](info): commencing operation
krb5kdc: starting...
W20260927 02:39:06.678298 392 mini_kdc.cc:121] Time spent starting KDC: real 1.216s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/krb5kdc/test-user.keytab.
Sep 27 02:39:06 dist-test-slave-zv8z krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:39:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:39:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:39:06.789738 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38209
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38209
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:39:06.894742 5946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:39:06.894996 5946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:39:06.895035 5946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:39:06.898973 5946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:39:06.899029 5946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:39:06.899045 5946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:39:06.899060 5946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:39:06.899075 5946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:39:06.904124 5946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:39:06.905048 5946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:39:06.905975 5946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:39:06.913826 5952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:06.913867 5954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:06.913826 5951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:06.914052 5946 server_base.cc:1034] running on GCE node
I20260927 02:39:06.914676 5946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:39:06.915742 5946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:39:06.916904 5946 hybrid_clock.cc:648] HybridClock initialized: now 1790476746916886 us; error 29 us; skew 500 ppm
Sep 27 02:39:06 dist-test-slave-zv8z krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:39:06.919575 5946 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:39:06.920368 5946 webserver.cc:460] Webserver started at http://127.0.98.62:38467/ using document root <none> and password file <none>
I20260927 02:39:06.920843 5946 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:06.920897 5946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:06.921061 5946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:39:06.922986 5946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/master-0/data/instance:
uuid: "03a5020ac68f4412986bd290b9ead976"
format_stamp: "Formatted at 2026-09-27 02:39:06 on dist-test-slave-zv8z"
server_key: "01d84f19d2466aadac8fc350445c537d"
server_key_iv: "c2dc1392916730267c0af2ee834816a8"
server_key_version: "encryptionkey@0"
I20260927 02:39:06.923472 5946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/master-0/wal/instance:
uuid: "03a5020ac68f4412986bd290b9ead976"
format_stamp: "Formatted at 2026-09-27 02:39:06 on dist-test-slave-zv8z"
server_key: "01d84f19d2466aadac8fc350445c537d"
server_key_iv: "c2dc1392916730267c0af2ee834816a8"
server_key_version: "encryptionkey@0"
I20260927 02:39:06.926749 5946 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:39:06.928973 5961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:06.930184 5946 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:39:06.930320 5946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "03a5020ac68f4412986bd290b9ead976"
format_stamp: "Formatted at 2026-09-27 02:39:06 on dist-test-slave-zv8z"
server_key: "01d84f19d2466aadac8fc350445c537d"
server_key_iv: "c2dc1392916730267c0af2ee834816a8"
server_key_version: "encryptionkey@0"
I20260927 02:39:06.930410 5946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:06.940847 5946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:39:06.943687 5946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:39:06.943847 5946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:39:06.949849 5946 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38209
I20260927 02:39:06.950098 6013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38209 every 8 connection(s)
I20260927 02:39:06.950773 5946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:39:06.953325 6014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:39:06.954934 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 5946
I20260927 02:39:06.955094 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/master-0/wal/instance
I20260927 02:39:06.955345 392 external_mini_cluster.cc:1428] Setting key 2bf26533f86c408786a5e97a6e767957
I20260927 02:39:06.959685 6014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03a5020ac68f4412986bd290b9ead976: Bootstrap starting.
I20260927 02:39:06.962146 6014 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03a5020ac68f4412986bd290b9ead976: Neither blocks nor log segments found. Creating new log.
I20260927 02:39:06.963006 6014 log.cc:824] T 00000000000000000000000000000000 P 03a5020ac68f4412986bd290b9ead976: Log is configured to *not* fsync() on all Append() calls
Sep 27 02:39:06 dist-test-slave-zv8z krb5kdc[5933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:39:06.964954 6014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03a5020ac68f4412986bd290b9ead976: No bootstrap required, opened a new log
I20260927 02:39:06.969310 6014 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03a5020ac68f4412986bd290b9ead976 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03a5020ac68f4412986bd290b9ead976" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38209 } }
I20260927 02:39:06.969556 6014 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03a5020ac68f4412986bd290b9ead976 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:39:06.969601 6014 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03a5020ac68f4412986bd290b9ead976 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03a5020ac68f4412986bd290b9ead976, State: Initialized, Role: FOLLOWER
I20260927 02:39:06.969761 6017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:06.956617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38662 (local address 127.0.98.62:38209)
0927 02:39:06.956997 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:06.957015 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:06.957055 (+ 40us) server_negotiation.cc:407] Connection header received
0927 02:39:06.957812 (+ 757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:06.957836 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:06.958240 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:06.958649 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:06.959301 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:06.960697 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:06.961583 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:06.962272 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:06.962457 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:06.964926 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:06.964932 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:06.964948 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:06.964977 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:06.966817 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:06.967214 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:06.967221 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:06.967227 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:06.967383 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:06.967648 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:06.967651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:06.967653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:06.968126 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:06.968316 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:06.968532 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 02:39:06.968832 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":95,"threads_started":1}
I20260927 02:39:06.970059 6014 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03a5020ac68f4412986bd290b9ead976 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03a5020ac68f4412986bd290b9ead976" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38209 } }
I20260927 02:39:06.970193 6014 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03a5020ac68f4412986bd290b9ead976 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:39:06.970244 6014 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03a5020ac68f4412986bd290b9ead976 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:39:06.970361 6014 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03a5020ac68f4412986bd290b9ead976 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:39:06.971333 6014 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03a5020ac68f4412986bd290b9ead976 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03a5020ac68f4412986bd290b9ead976" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38209 } }
I20260927 02:39:06.971673 6014 leader_election.cc:302] T 00000000000000000000000000000000 P 03a5020ac68f4412986bd290b9ead976 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03a5020ac68f4412986bd290b9ead976; no voters:
I20260927 02:39:06.971975 6014 leader_election.cc:288] T 00000000000000000000000000000000 P 03a5020ac68f4412986bd290b9ead976 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:39:06.972056 6019 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03a5020ac68f4412986bd290b9ead976 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:39:06.972285 6019 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03a5020ac68f4412986bd290b9ead976 [term 1 LEADER]: Becoming Leader. State: Replica: 03a5020ac68f4412986bd290b9ead976, State: Running, Role: LEADER
I20260927 02:39:06.972560 6019 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03a5020ac68f4412986bd290b9ead976 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03a5020ac68f4412986bd290b9ead976" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38209 } }
I20260927 02:39:06.973255 6014 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03a5020ac68f4412986bd290b9ead976 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:39:06.974419 6021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03a5020ac68f4412986bd290b9ead976 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03a5020ac68f4412986bd290b9ead976. Latest consensus state: current_term: 1 leader_uuid: "03a5020ac68f4412986bd290b9ead976" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03a5020ac68f4412986bd290b9ead976" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38209 } } }
I20260927 02:39:06.974551 6021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03a5020ac68f4412986bd290b9ead976 [sys.catalog]: This master's current role is: LEADER
I20260927 02:39:06.974956 6028 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:39:06.975947 6020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03a5020ac68f4412986bd290b9ead976 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03a5020ac68f4412986bd290b9ead976" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03a5020ac68f4412986bd290b9ead976" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38209 } } }
I20260927 02:39:06.976040 6020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03a5020ac68f4412986bd290b9ead976 [sys.catalog]: This master's current role is: LEADER
I20260927 02:39:06.977568 6028 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:39:06.982961 6028 catalog_manager.cc:1347] Generated new cluster ID: 2ca113d4c0a24b92b59c49ee2d4c7eba
I20260927 02:39:06.983016 6028 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:39:06.994109 6028 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:39:06.994989 6028 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:39:07.002727 6028 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03a5020ac68f4412986bd290b9ead976: Generated new TSK 0
I20260927 02:39:07.003410 6028 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:39:07.035590 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38209
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:39:07.142215 6040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:39:07.142475 6040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:39:07.142516 6040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:39:07.146541 6040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:39:07.146601 6040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:39:07.146660 6040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:39:07.151846 6040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:39:07.152863 6040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:39:07.153839 6040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:39:07.163055 6045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:07.163070 6046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:07.163087 6048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:07.163412 6040 server_base.cc:1034] running on GCE node
I20260927 02:39:07.163826 6040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:39:07.164446 6040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:39:07.165648 6040 hybrid_clock.cc:648] HybridClock initialized: now 1790476747165622 us; error 35 us; skew 500 ppm
Sep 27 02:39:07 dist-test-slave-zv8z krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476747, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:39:07.168483 6040 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:39:07.169404 6040 webserver.cc:460] Webserver started at http://127.0.98.1:38627/ using document root <none> and password file <none>
I20260927 02:39:07.169919 6040 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:07.169979 6040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:07.170171 6040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:39:07.172261 6040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/ts-0/data/instance:
uuid: "8c902f7006f64b8e9927bfa7c79c503b"
format_stamp: "Formatted at 2026-09-27 02:39:07 on dist-test-slave-zv8z"
server_key: "fbf395f7cd718f7e17c2b56087075a92"
server_key_iv: "4b57fca629ba55d76fb6ab6af24de00f"
server_key_version: "encryptionkey@0"
I20260927 02:39:07.172775 6040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8c902f7006f64b8e9927bfa7c79c503b"
format_stamp: "Formatted at 2026-09-27 02:39:07 on dist-test-slave-zv8z"
server_key: "fbf395f7cd718f7e17c2b56087075a92"
server_key_iv: "4b57fca629ba55d76fb6ab6af24de00f"
server_key_version: "encryptionkey@0"
I20260927 02:39:07.176672 6040 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 02:39:07.179080 6055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:07.180393 6040 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:39:07.180492 6040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/ts-0/wal
uuid: "8c902f7006f64b8e9927bfa7c79c503b"
format_stamp: "Formatted at 2026-09-27 02:39:07 on dist-test-slave-zv8z"
server_key: "fbf395f7cd718f7e17c2b56087075a92"
server_key_iv: "4b57fca629ba55d76fb6ab6af24de00f"
server_key_version: "encryptionkey@0"
I20260927 02:39:07.180590 6040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:07.200174 6040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:39:07.203109 6040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:39:07.203262 6040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:39:07.203850 6040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:39:07.204653 6040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:39:07.204698 6040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:07.204738 6040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:39:07.204766 6040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:07.213519 6040 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43703
I20260927 02:39:07.213541 6168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43703 every 8 connection(s)
I20260927 02:39:07.214429 6040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:39:07.221387 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 6040
I20260927 02:39:07.221495 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:39:07.221779 392 external_mini_cluster.cc:1428] Setting key d1d9bfdde75ba5543de89f4aad2d70b8
Sep 27 02:39:07 dist-test-slave-zv8z krb5kdc[5933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476747, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:39:07.228135 6017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:07.216230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37559 (local address 127.0.98.62:38209)
0927 02:39:07.216388 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:07.216393 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:07.217325 (+ 932us) server_negotiation.cc:407] Connection header received
0927 02:39:07.218690 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:07.218694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:07.218740 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:07.218812 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:07.219983 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:07.220513 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:07.221739 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:07.222052 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:07.222167 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:07.224765 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:07.224768 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:07.224771 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:07.224793 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:07.226240 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:07.226721 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:07.226724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:07.226726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:07.226805 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:07.227106 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:07.227109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:07.227111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:07.227284 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:07.227367 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:07.227840 (+ 473us) server_negotiation.cc:299] Negotiation successful
0927 02:39:07.227970 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 02:39:07.228915 6171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:07.216687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38209 (local address 127.0.98.1:37559)
0927 02:39:07.217172 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:07.217218 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:07.218459 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:07.218966 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:07.218972 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:07.219390 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:07.219858 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:07.219870 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:07.220635 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:07.220638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:07.221627 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:07.221637 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:07.222234 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:07.222239 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:07.222452 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:07.223013 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:07.223034 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:07.224611 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:07.226363 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:07.226373 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:07.226388 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:07.226631 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:07.226887 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:07.226891 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:07.226893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:07.227022 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:07.227386 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:07.227392 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:07.227622 (+ 230us) client_negotiation.cc:769] Sending connection context
0927 02:39:07.227785 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 02:39:07.228101 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":111,"threads_started":1}
I20260927 02:39:07.230232 6169 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38209
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:39:07.230482 6169 heartbeater.cc:464] Registering TS with master...
I20260927 02:39:07.230958 6169 heartbeater.cc:511] Master 127.0.98.62:38209 requested a full tablet report, sending...
I20260927 02:39:07.232597 5978 ts_manager.cc:194] Registered new tserver with Master: 8c902f7006f64b8e9927bfa7c79c503b (127.0.98.1:43703)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 02:39:07.233953 5978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37559
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:39:07.249228 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38209
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:39:07.355295 6174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:39:07.355544 6174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:39:07.355585 6174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:39:07.359642 6174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:39:07.359709 6174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:39:07.359782 6174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:39:07.364966 6174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:39:07.365888 6174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:39:07.366888 6174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:39:07.376183 6180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:07.376255 6182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:07.376185 6179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:07.376405 6174 server_base.cc:1034] running on GCE node
I20260927 02:39:07.376947 6174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:39:07.377568 6174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:39:07.378724 6174 hybrid_clock.cc:648] HybridClock initialized: now 1790476747378706 us; error 34 us; skew 500 ppm
Sep 27 02:39:07 dist-test-slave-zv8z krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476747, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:39:07.381448 6174 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:39:07.382433 6174 webserver.cc:460] Webserver started at http://127.0.98.2:37861/ using document root <none> and password file <none>
I20260927 02:39:07.382972 6174 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:07.383036 6174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:07.383244 6174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:39:07.385401 6174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/ts-1/data/instance:
uuid: "74e2379bda944603ad6056abb056e729"
format_stamp: "Formatted at 2026-09-27 02:39:07 on dist-test-slave-zv8z"
server_key: "12be062852b545a5f2fa372fafa22fa0"
server_key_iv: "08ae0f5bf130d1ddba5b31da06a3c91b"
server_key_version: "encryptionkey@0"
I20260927 02:39:07.385947 6174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/ts-1/wal/instance:
uuid: "74e2379bda944603ad6056abb056e729"
format_stamp: "Formatted at 2026-09-27 02:39:07 on dist-test-slave-zv8z"
server_key: "12be062852b545a5f2fa372fafa22fa0"
server_key_iv: "08ae0f5bf130d1ddba5b31da06a3c91b"
server_key_version: "encryptionkey@0"
I20260927 02:39:07.389832 6174 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260927 02:39:07.392310 6189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:07.393517 6174 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:39:07.393622 6174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/ts-1/wal
uuid: "74e2379bda944603ad6056abb056e729"
format_stamp: "Formatted at 2026-09-27 02:39:07 on dist-test-slave-zv8z"
server_key: "12be062852b545a5f2fa372fafa22fa0"
server_key_iv: "08ae0f5bf130d1ddba5b31da06a3c91b"
server_key_version: "encryptionkey@0"
I20260927 02:39:07.393726 6174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:07.427656 6174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:39:07.430541 6174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:39:07.430712 6174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:39:07.431279 6174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:39:07.432060 6174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:39:07.432106 6174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:07.432157 6174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:39:07.432183 6174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:07.440315 6174 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33791
I20260927 02:39:07.440436 6302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33791 every 8 connection(s)
I20260927 02:39:07.441155 6174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:39:07.445207 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 6174
I20260927 02:39:07.445333 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:39:07.445582 392 external_mini_cluster.cc:1428] Setting key 38942c02789f6f8fd8d01d058588058a
Sep 27 02:39:07 dist-test-slave-zv8z krb5kdc[5933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476747, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:39:07.454577 6017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:07.442832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40649 (local address 127.0.98.62:38209)
0927 02:39:07.442961 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:07.442965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:07.443808 (+ 843us) server_negotiation.cc:407] Connection header received
0927 02:39:07.445088 (+ 1280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:07.445091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:07.445144 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:07.445224 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:07.446521 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:07.447037 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:07.448149 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:07.448371 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:07.448457 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:07.451109 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:07.451112 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:07.451114 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:07.451137 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:07.452397 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:07.452976 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:07.452979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:07.452980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:07.453063 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:07.453439 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:07.453442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:07.453443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:07.453615 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:07.453702 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:07.454332 (+ 630us) server_negotiation.cc:299] Negotiation successful
0927 02:39:07.454434 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 02:39:07.455338 6305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:07.443230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38209 (local address 127.0.98.2:40649)
0927 02:39:07.443646 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:07.443686 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:07.444890 (+ 1204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:07.445419 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:07.445429 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:07.445801 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:07.446387 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:07.446403 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:07.447156 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:07.447159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:07.448032 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:07.448041 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:07.448502 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:07.448506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:07.448662 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:07.449261 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:07.449277 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:07.450947 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:07.452507 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:07.452513 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:07.452529 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:07.452860 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:07.453157 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:07.453160 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:07.453162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:07.453350 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:07.453721 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:07.453727 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:07.454032 (+ 305us) client_negotiation.cc:769] Sending connection context
0927 02:39:07.454245 (+ 213us) client_negotiation.cc:240] Negotiation successful
0927 02:39:07.454536 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":109,"threads_started":1}
I20260927 02:39:07.456405 6303 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38209
I20260927 02:39:07.456650 6303 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 02:39:07.457228 6303 heartbeater.cc:511] Master 127.0.98.62:38209 requested a full tablet report, sending...
I20260927 02:39:07.458323 5978 ts_manager.cc:194] Registered new tserver with Master: 74e2379bda944603ad6056abb056e729 (127.0.98.2:33791)
I20260927 02:39:07.459055 5978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40649
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:39:07.471371 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38209
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:39:07.576313 6308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:39:07.576587 6308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:39:07.576650 6308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:39:07.580693 6308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:39:07.580755 6308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:39:07.580835 6308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:39:07.586087 6308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:39:07.587093 6308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:39:07.588063 6308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:39:07.597091 6314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:07.597126 6313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:07.597214 6308 server_base.cc:1034] running on GCE node
W20260927 02:39:07.597312 6316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:07.597769 6308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:39:07.598415 6308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:39:07.599579 6308 hybrid_clock.cc:648] HybridClock initialized: now 1790476747599550 us; error 40 us; skew 500 ppm
Sep 27 02:39:07 dist-test-slave-zv8z krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476747, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:39:07.602208 6308 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:39:07.603037 6308 webserver.cc:460] Webserver started at http://127.0.98.3:43817/ using document root <none> and password file <none>
I20260927 02:39:07.603547 6308 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:07.603605 6308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:07.603796 6308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:39:07.605705 6308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/ts-2/data/instance:
uuid: "53c235b7c9034f31b3ac447b24d135f1"
format_stamp: "Formatted at 2026-09-27 02:39:07 on dist-test-slave-zv8z"
server_key: "833dfa256823961a4e18942c64be32a0"
server_key_iv: "3423fcafd09889cb37692f1bae908c01"
server_key_version: "encryptionkey@0"
I20260927 02:39:07.606215 6308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/ts-2/wal/instance:
uuid: "53c235b7c9034f31b3ac447b24d135f1"
format_stamp: "Formatted at 2026-09-27 02:39:07 on dist-test-slave-zv8z"
server_key: "833dfa256823961a4e18942c64be32a0"
server_key_iv: "3423fcafd09889cb37692f1bae908c01"
server_key_version: "encryptionkey@0"
I20260927 02:39:07.609639 6308 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 02:39:07.611876 6323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:07.613011 6308 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:39:07.613108 6308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/ts-2/wal
uuid: "53c235b7c9034f31b3ac447b24d135f1"
format_stamp: "Formatted at 2026-09-27 02:39:07 on dist-test-slave-zv8z"
server_key: "833dfa256823961a4e18942c64be32a0"
server_key_iv: "3423fcafd09889cb37692f1bae908c01"
server_key_version: "encryptionkey@0"
I20260927 02:39:07.613256 6308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:07.637554 6308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:39:07.640586 6308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:39:07.640759 6308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:39:07.641320 6308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:39:07.642127 6308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:39:07.642180 6308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:07.642230 6308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:39:07.642278 6308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:07.651124 6308 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37011
I20260927 02:39:07.651149 6436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37011 every 8 connection(s)
I20260927 02:39:07.652006 6308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:39:07.656920 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 6308
I20260927 02:39:07.657042 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:39:07.657352 392 external_mini_cluster.cc:1428] Setting key a917d00f4209bc306432be064e94188a
Sep 27 02:39:07 dist-test-slave-zv8z krb5kdc[5933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476747, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:39:07.666982 6017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:07.653671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37425 (local address 127.0.98.62:38209)
0927 02:39:07.653804 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:07.653809 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:07.654851 (+ 1042us) server_negotiation.cc:407] Connection header received
0927 02:39:07.656579 (+ 1728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:07.656584 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:07.656643 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:07.656740 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:07.658501 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:07.659288 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:07.660715 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:07.661029 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:07.661123 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:07.663660 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:07.663663 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:07.663665 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:07.663690 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:07.665018 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:07.665616 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:07.665619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:07.665620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:07.665701 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:07.666024 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:07.666026 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:07.666027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:07.666242 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:07.666356 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:07.666776 (+ 420us) server_negotiation.cc:299] Negotiation successful
0927 02:39:07.666868 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 02:39:07.667815 6439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:07.654133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38209 (local address 127.0.98.3:37425)
0927 02:39:07.654698 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:07.654756 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:07.656327 (+ 1571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:07.657061 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:07.657072 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:07.657642 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:07.658333 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:07.658355 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:07.659432 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:07.659435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:07.660598 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:07.660610 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:07.661167 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:07.661172 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:07.661362 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:07.661993 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:07.662010 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:07.663495 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:07.665211 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:07.665216 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:07.665237 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:07.665495 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:07.665803 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:07.665806 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:07.665807 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:07.665942 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:07.666353 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:07.666358 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:07.666582 (+ 224us) client_negotiation.cc:769] Sending connection context
0927 02:39:07.666732 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 02:39:07.666973 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":140,"threads_started":1}
I20260927 02:39:07.668883 6437 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38209
I20260927 02:39:07.669077 6437 heartbeater.cc:464] Registering TS with master...
I20260927 02:39:07.669488 6437 heartbeater.cc:511] Master 127.0.98.62:38209 requested a full tablet report, sending...
I20260927 02:39:07.670605 5977 ts_manager.cc:194] Registered new tserver with Master: 53c235b7c9034f31b3ac447b24d135f1 (127.0.98.3:37011)
I20260927 02:39:07.671278 5977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37425
I20260927 02:39:07.671392 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/krb5kdc/test-admin.keytab.
Sep 27 02:39:07 dist-test-slave-zv8z krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:39:07.686234 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 02:39:07.688800 6444 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:07.687509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57508 (local address 127.0.98.1:43703)
0927 02:39:07.687821 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:07.687833 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:07.687855 (+ 22us) server_negotiation.cc:407] Connection header received
0927 02:39:07.687902 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:07.687907 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:07.688076 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:07.688187 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:07.688650 (+ 463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57508: BlockingRecv error: recv got EOF from 127.0.0.1:57508 (error 108)
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":131,"threads_started":1}
W20260927 02:39:07.688838 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 02:39:07 dist-test-slave-zv8z krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:39:07.689749 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 02:39:07 dist-test-slave-zv8z krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:39:07.690614 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 02:39:07 dist-test-slave-zv8z krb5kdc[5933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:39:07.699636 6444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:07.691378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57522 (local address 127.0.98.1:43703)
0927 02:39:07.691506 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:07.691509 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:07.691558 (+ 49us) server_negotiation.cc:407] Connection header received
0927 02:39:07.691680 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:07.691683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:07.691725 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:07.691791 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:07.692266 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:07.693023 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:07.693882 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:07.694278 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:07.694366 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:07.696520 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:07.696530 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:07.696546 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:07.696572 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:07.697904 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:07.698297 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:07.698302 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:07.698304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:07.698376 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:07.698667 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:07.698670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:07.698671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:07.698902 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:07.699066 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:07.699327 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 02:39:07.699498 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 02:39:07.703122 6444 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:07.702237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57530 (local address 127.0.98.1:43703)
0927 02:39:07.702398 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:07.702401 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:07.702559 (+ 158us) server_negotiation.cc:407] Connection header received
0927 02:39:07.702640 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:07.702642 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:07.702677 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:07.702747 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:07.703031 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57530: BlockingRecv error: recv got EOF from 127.0.0.1:57530 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260927 02:39:07.703279 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 02:39:07 dist-test-slave-zv8z krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:39:07.704213 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 02:39:07 dist-test-slave-zv8z krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:39:07.705035 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 02:39:07 dist-test-slave-zv8z krb5kdc[5933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:39:07.712589 6444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:07.705740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57542 (local address 127.0.98.1:43703)
0927 02:39:07.705828 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:07.705832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:07.705948 (+ 116us) server_negotiation.cc:407] Connection header received
0927 02:39:07.706041 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:07.706044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:07.706078 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:07.706135 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:07.706637 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:07.707122 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:07.707853 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:07.708066 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:07.708147 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:07.710112 (+ 1965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:07.710115 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:07.710116 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:07.710137 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:07.711210 (+ 1073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:07.711605 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:07.711609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:07.711610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:07.711678 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:07.711907 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:07.711910 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:07.711912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:07.712054 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:07.712158 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:07.712366 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 02:39:07.712459 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260927 02:39:07.714684 6444 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:07.713930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57548 (local address 127.0.98.1:43703)
0927 02:39:07.714044 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:07.714047 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:07.714096 (+ 49us) server_negotiation.cc:407] Connection header received
0927 02:39:07.714185 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:07.714187 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:07.714217 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:07.714336 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:07.714601 (+ 265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57548: BlockingRecv error: recv got EOF from 127.0.0.1:57548 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 02:39:07.714787 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790476718023131-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 02:39:07 dist-test-slave-zv8z krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:39:07.715642 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 02:39:07 dist-test-slave-zv8z krb5kdc[5933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:39:07.716385 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 02:39:07 dist-test-slave-zv8z krb5kdc[5933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:39:07.724035 6444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:07.717120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57564 (local address 127.0.98.1:43703)
0927 02:39:07.717230 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:07.717234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:07.717336 (+ 102us) server_negotiation.cc:407] Connection header received
0927 02:39:07.717432 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:07.717435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:07.717469 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:07.717530 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:07.717953 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:07.718491 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:07.719223 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:07.719431 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:07.719512 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:07.721481 (+ 1969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:07.721484 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:07.721486 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:07.721506 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:07.722670 (+ 1164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:07.723038 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:07.723041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:07.723043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:07.723108 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:07.723340 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:07.723343 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:07.723345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:07.723490 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:07.723595 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:07.723811 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 02:39:07.723905 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 02:39:07.724920 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 6040
I20260927 02:39:07.729668 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 6174
I20260927 02:39:07.734315 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 6308
I20260927 02:39:07.738404 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 5946
2026-09-27T02:39:07Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2295 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:39:07 dist-test-slave-zv8z krb5kdc[6457](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:39:07 dist-test-slave-zv8z krb5kdc[6457](info): set up 2 sockets
Sep 27 02:39:07 dist-test-slave-zv8z krb5kdc[6457](info): commencing operation
krb5kdc: starting...
W20260927 02:39:08.690485 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/test-user.keytab.
Sep 27 02:39:08 dist-test-slave-zv8z krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:39:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:39:08Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260927 02:39:08.804620 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35857
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35857
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:39:08.908690 6470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:39:08.908942 6470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:39:08.908980 6470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:39:08.912994 6470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:39:08.913051 6470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:39:08.913069 6470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:39:08.913084 6470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:39:08.913097 6470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:39:08.918311 6470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:39:08.919272 6470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:39:08.920215 6470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:39:08.927991 6475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:08.928107 6470 server_base.cc:1034] running on GCE node
W20260927 02:39:08.928037 6476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:08.928010 6478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:08.928727 6470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:39:08.929720 6470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:39:08.930859 6470 hybrid_clock.cc:648] HybridClock initialized: now 1790476748930839 us; error 37 us; skew 500 ppm
Sep 27 02:39:08 dist-test-slave-zv8z krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476748, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:39:08.933359 6470 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260927 02:39:08.934185 6470 webserver.cc:460] Webserver started at http://127.0.98.62:34655/ using document root <none> and password file <none>
I20260927 02:39:08.934710 6470 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:08.934772 6470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:08.934947 6470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:39:08.936813 6470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-0/data/instance:
uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f"
format_stamp: "Formatted at 2026-09-27 02:39:08 on dist-test-slave-zv8z"
server_key: "8cdcead5e7fec39a59052601ac429702"
server_key_iv: "54b210df7c8d55506ebca7eeacb721b7"
server_key_version: "encryptionkey@0"
I20260927 02:39:08.937311 6470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-0/wal/instance:
uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f"
format_stamp: "Formatted at 2026-09-27 02:39:08 on dist-test-slave-zv8z"
server_key: "8cdcead5e7fec39a59052601ac429702"
server_key_iv: "54b210df7c8d55506ebca7eeacb721b7"
server_key_version: "encryptionkey@0"
I20260927 02:39:08.940707 6470 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 02:39:08.942898 6485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:08.944054 6470 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:39:08.944159 6470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f"
format_stamp: "Formatted at 2026-09-27 02:39:08 on dist-test-slave-zv8z"
server_key: "8cdcead5e7fec39a59052601ac429702"
server_key_iv: "54b210df7c8d55506ebca7eeacb721b7"
server_key_version: "encryptionkey@0"
I20260927 02:39:08.944242 6470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:08.962347 6470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:39:08.964957 6470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:39:08.965123 6470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:39:08.971490 6470 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35857
I20260927 02:39:08.971743 6537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35857 every 8 connection(s)
I20260927 02:39:08.972357 6470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:39:08.974921 6538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:39:08.979864 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 6470
I20260927 02:39:08.980000 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-0/wal/instance
I20260927 02:39:08.980294 392 external_mini_cluster.cc:1428] Setting key a6f6c0ffcdd4e9b0732f0c2b8668bd28
I20260927 02:39:08.980291 6538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f: Bootstrap starting.
I20260927 02:39:08.982620 6538 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f: Neither blocks nor log segments found. Creating new log.
I20260927 02:39:08.983337 6538 log.cc:824] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f: Log is configured to *not* fsync() on all Append() calls
I20260927 02:39:08.985132 6538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f: No bootstrap required, opened a new log
Sep 27 02:39:08 dist-test-slave-zv8z krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:39:08.989018 6538 raft_consensus.cc:348] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } }
I20260927 02:39:08.989225 6538 raft_consensus.cc:374] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:39:08.989279 6538 raft_consensus.cc:729] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 372b9126cbeb4dbdbd3b50c5bfa23a8f, State: Initialized, Role: FOLLOWER
I20260927 02:39:08.989701 6538 consensus_queue.cc:260] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } }
I20260927 02:39:08.989856 6538 raft_consensus.cc:388] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:39:08.989929 6538 raft_consensus.cc:482] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:39:08.990021 6538 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:39:08.990855 6538 raft_consensus.cc:504] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } }
I20260927 02:39:08.991204 6538 leader_election.cc:302] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 372b9126cbeb4dbdbd3b50c5bfa23a8f; no voters:
I20260927 02:39:08.991513 6538 leader_election.cc:288] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:39:08.991607 6543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:39:08.991813 6543 raft_consensus.cc:686] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f [term 1 LEADER]: Becoming Leader. State: Replica: 372b9126cbeb4dbdbd3b50c5bfa23a8f, State: Running, Role: LEADER
I20260927 02:39:08.992182 6543 consensus_queue.cc:237] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } }
I20260927 02:39:08.992682 6538 sys_catalog.cc:564] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:39:08.993377 6545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 372b9126cbeb4dbdbd3b50c5bfa23a8f. Latest consensus state: current_term: 1 leader_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } } }
I20260927 02:39:08.993508 6545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f [sys.catalog]: This master's current role is: LEADER
I20260927 02:39:08.993871 6550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:39:08.994161 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:08.981493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39332 (local address 127.0.98.62:35857)
0927 02:39:08.981867 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:08.981882 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:08.981918 (+ 36us) server_negotiation.cc:407] Connection header received
0927 02:39:08.982568 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:08.982585 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:08.982925 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:08.983192 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:08.983993 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:08.984966 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:08.986048 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:08.986567 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:08.986681 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:08.989215 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:08.989220 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:08.989233 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:08.989261 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:08.990969 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:08.991413 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:08.991419 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:08.991423 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:08.991558 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:08.991883 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:08.991887 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:08.991888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:08.992382 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:08.992559 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:08.992866 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 02:39:08.993166 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":98,"threads_started":1}
I20260927 02:39:08.994143 6544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } } }
I20260927 02:39:08.994227 6544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f [sys.catalog]: This master's current role is: LEADER
I20260927 02:39:08.996912 6550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:39:09.002547 6550 catalog_manager.cc:1347] Generated new cluster ID: 7432986bc1b74b229d125db051eba32d
I20260927 02:39:09.002610 6550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:39:09.009249 6550 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:39:09.010082 6550 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:39:09.014122 6550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f: Generated new TSK 0
I20260927 02:39:09.014794 6550 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260927 02:39:09.043272 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35857
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:39:09.153595 6564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:39:09.153930 6564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:39:09.154017 6564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:39:09.159003 6564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:39:09.159076 6564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:39:09.159161 6564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:39:09.166159 6564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:39:09.167263 6564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:39:09.168323 6564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:39:09.177747 6572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:09.177747 6570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:09.177780 6569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:09.178193 6564 server_base.cc:1034] running on GCE node
I20260927 02:39:09.178670 6564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:39:09.179456 6564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:39:09.180630 6564 hybrid_clock.cc:648] HybridClock initialized: now 1790476749180613 us; error 32 us; skew 500 ppm
Sep 27 02:39:09 dist-test-slave-zv8z krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476749, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:39:09.183390 6564 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 02:39:09.184237 6564 webserver.cc:460] Webserver started at http://127.0.98.1:37125/ using document root <none> and password file <none>
I20260927 02:39:09.184815 6564 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:09.184880 6564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:09.185081 6564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:39:09.187170 6564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-0/data/instance:
uuid: "6c73821393684ab0961970fdda44e818"
format_stamp: "Formatted at 2026-09-27 02:39:09 on dist-test-slave-zv8z"
server_key: "b5eb7b8b2e1d5497368435e62c6be8f8"
server_key_iv: "b179f0e41004e42dbebe3dbb02b16509"
server_key_version: "encryptionkey@0"
I20260927 02:39:09.187685 6564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6c73821393684ab0961970fdda44e818"
format_stamp: "Formatted at 2026-09-27 02:39:09 on dist-test-slave-zv8z"
server_key: "b5eb7b8b2e1d5497368435e62c6be8f8"
server_key_iv: "b179f0e41004e42dbebe3dbb02b16509"
server_key_version: "encryptionkey@0"
I20260927 02:39:09.191232 6564 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 02:39:09.193542 6579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:09.194830 6564 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:39:09.194947 6564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-0/wal
uuid: "6c73821393684ab0961970fdda44e818"
format_stamp: "Formatted at 2026-09-27 02:39:09 on dist-test-slave-zv8z"
server_key: "b5eb7b8b2e1d5497368435e62c6be8f8"
server_key_iv: "b179f0e41004e42dbebe3dbb02b16509"
server_key_version: "encryptionkey@0"
I20260927 02:39:09.195047 6564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:09.208369 6564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:39:09.211162 6564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:39:09.211308 6564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:39:09.211814 6564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:39:09.212560 6564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:39:09.212608 6564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:09.212646 6564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:39:09.212662 6564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:09.220649 6564 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35281
I20260927 02:39:09.220700 6692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35281 every 8 connection(s)
I20260927 02:39:09.221482 6564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:39:09.228770 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 6564
I20260927 02:39:09.228884 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:39:09.229122 392 external_mini_cluster.cc:1428] Setting key 9fc151a104377ebd1cae1fcc0641c2d2
Sep 27 02:39:09 dist-test-slave-zv8z krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476749, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:39:09.234324 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:09.223129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48875 (local address 127.0.98.62:35857)
0927 02:39:09.223251 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:09.223254 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:09.224079 (+ 825us) server_negotiation.cc:407] Connection header received
0927 02:39:09.225509 (+ 1430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:09.225512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:09.225557 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:09.225628 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:09.226748 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:09.227268 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:09.228305 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:09.228557 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:09.228647 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:09.231120 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:09.231124 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:09.231127 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:09.231153 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:09.232458 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:09.232909 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:09.232913 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:09.232914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:09.232993 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:09.233289 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:09.233292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:09.233294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:09.233463 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:09.233547 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:09.234014 (+ 467us) server_negotiation.cc:299] Negotiation successful
0927 02:39:09.234132 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 02:39:09.235091 6695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:09.223490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35857 (local address 127.0.98.1:48875)
0927 02:39:09.223913 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:09.223968 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:09.225329 (+ 1361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:09.225775 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:09.225781 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:09.226114 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:09.226626 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:09.226640 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:09.227363 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:09.227365 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:09.228219 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:09.228227 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:09.228681 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:09.228683 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:09.228812 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:09.229387 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:09.229406 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:09.230971 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:09.232562 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:09.232567 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:09.232581 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:09.232817 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:09.233074 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:09.233076 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:09.233078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:09.233206 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:09.233558 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:09.233564 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:09.233795 (+ 231us) client_negotiation.cc:769] Sending connection context
0927 02:39:09.233955 (+ 160us) client_negotiation.cc:240] Negotiation successful
0927 02:39:09.234299 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":106,"threads_started":1}
I20260927 02:39:09.236217 6693 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35857
I20260927 02:39:09.236426 6693 heartbeater.cc:464] Registering TS with master...
I20260927 02:39:09.236871 6693 heartbeater.cc:511] Master 127.0.98.62:35857 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:39:09.238325 6502 ts_manager.cc:194] Registered new tserver with Master: 6c73821393684ab0961970fdda44e818 (127.0.98.1:35281)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260927 02:39:09.239711 6502 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48875
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260927 02:39:09.253620 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35857
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:39:09.359318 6698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:39:09.359571 6698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:39:09.359612 6698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:39:09.363565 6698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:39:09.363620 6698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:39:09.363680 6698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:39:09.368813 6698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:39:09.369807 6698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:39:09.370814 6698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:39:09.380090 6703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:09.380090 6704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:09.380153 6706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:09.380316 6698 server_base.cc:1034] running on GCE node
I20260927 02:39:09.380843 6698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:39:09.381477 6698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:39:09.382632 6698 hybrid_clock.cc:648] HybridClock initialized: now 1790476749382617 us; error 31 us; skew 500 ppm
Sep 27 02:39:09 dist-test-slave-zv8z krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476749, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:39:09.385207 6698 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 02:39:09.386057 6698 webserver.cc:460] Webserver started at http://127.0.98.2:41865/ using document root <none> and password file <none>
I20260927 02:39:09.386628 6698 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:09.386696 6698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:09.386897 6698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:39:09.388976 6698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-1/data/instance:
uuid: "bfb91636c48348d1abc15549cd284922"
format_stamp: "Formatted at 2026-09-27 02:39:09 on dist-test-slave-zv8z"
server_key: "46878191608bcfef1679238110e2ca4f"
server_key_iv: "1a7ca4173c58b856141f965d276b204e"
server_key_version: "encryptionkey@0"
I20260927 02:39:09.389530 6698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bfb91636c48348d1abc15549cd284922"
format_stamp: "Formatted at 2026-09-27 02:39:09 on dist-test-slave-zv8z"
server_key: "46878191608bcfef1679238110e2ca4f"
server_key_iv: "1a7ca4173c58b856141f965d276b204e"
server_key_version: "encryptionkey@0"
I20260927 02:39:09.393371 6698 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 02:39:09.395947 6713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:09.397158 6698 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:39:09.397266 6698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-1/wal
uuid: "bfb91636c48348d1abc15549cd284922"
format_stamp: "Formatted at 2026-09-27 02:39:09 on dist-test-slave-zv8z"
server_key: "46878191608bcfef1679238110e2ca4f"
server_key_iv: "1a7ca4173c58b856141f965d276b204e"
server_key_version: "encryptionkey@0"
I20260927 02:39:09.397364 6698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:09.408588 6698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:39:09.411267 6698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:39:09.411434 6698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:39:09.411968 6698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:39:09.412732 6698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:39:09.412776 6698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:09.412827 6698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:39:09.412856 6698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:09.421229 6698 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38069
I20260927 02:39:09.421301 6826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38069 every 8 connection(s)
I20260927 02:39:09.422103 6698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:39:09.429342 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 6698
I20260927 02:39:09.429482 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:39:09.429741 392 external_mini_cluster.cc:1428] Setting key 6cadabbb4aa1e5c53c5309ab3ac8e065
Sep 27 02:39:09 dist-test-slave-zv8z krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476749, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:39:09.435858 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:09.423830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38997 (local address 127.0.98.62:35857)
0927 02:39:09.423964 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:09.423968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:09.424715 (+ 747us) server_negotiation.cc:407] Connection header received
0927 02:39:09.426010 (+ 1295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:09.426014 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:09.426071 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:09.426145 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:09.427410 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:09.427931 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:09.429023 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:09.429239 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:09.429327 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:09.432323 (+ 2996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:09.432327 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:09.432330 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:09.432358 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:09.433740 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:09.434329 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:09.434331 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:09.434333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:09.434408 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:09.434743 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:09.434745 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:09.434746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:09.434944 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:09.435017 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:09.435641 (+ 624us) server_negotiation.cc:299] Negotiation successful
0927 02:39:09.435739 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 02:39:09.436992 6829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:09.424125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35857 (local address 127.0.98.2:38997)
0927 02:39:09.424570 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:09.424611 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:09.425809 (+ 1198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:09.426421 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:09.426428 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:09.426771 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:09.427258 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:09.427272 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:09.428047 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:09.428051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:09.428909 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:09.428919 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:09.429378 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:09.429382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:09.429578 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:09.430134 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:09.430150 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:09.432157 (+ 2007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:09.433855 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:09.433861 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:09.433877 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:09.434192 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:09.434487 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:09.434490 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:09.434492 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:09.434660 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:09.435047 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:09.435054 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:09.435376 (+ 322us) client_negotiation.cc:769] Sending connection context
0927 02:39:09.435576 (+ 200us) client_negotiation.cc:240] Negotiation successful
0927 02:39:09.435924 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":108,"threads_started":1}
I20260927 02:39:09.438122 6827 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35857
I20260927 02:39:09.438376 6827 heartbeater.cc:464] Registering TS with master...
I20260927 02:39:09.438827 6827 heartbeater.cc:511] Master 127.0.98.62:35857 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:39:09.439826 6502 ts_manager.cc:194] Registered new tserver with Master: bfb91636c48348d1abc15549cd284922 (127.0.98.2:38069)
I20260927 02:39:09.440486 6502 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38997
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 02:39:09.456506 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35857
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:39:09.562520 6832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:39:09.562762 6832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:39:09.562819 6832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:39:09.567013 6832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:39:09.567080 6832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:39:09.567159 6832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:39:09.572358 6832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:39:09.573328 6832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:39:09.574347 6832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:39:09.583238 6832 server_base.cc:1034] running on GCE node
W20260927 02:39:09.583179 6837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:09.583189 6840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:09.583177 6838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:09.583765 6832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:39:09.584359 6832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:39:09.585520 6832 hybrid_clock.cc:648] HybridClock initialized: now 1790476749585495 us; error 34 us; skew 500 ppm
Sep 27 02:39:09 dist-test-slave-zv8z krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476749, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:39:09.588150 6832 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 02:39:09.588963 6832 webserver.cc:460] Webserver started at http://127.0.98.3:37539/ using document root <none> and password file <none>
I20260927 02:39:09.589493 6832 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:09.589565 6832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:09.589741 6832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:39:09.591732 6832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-2/data/instance:
uuid: "95c9bea2bd4044729172198bd0ee9bd4"
format_stamp: "Formatted at 2026-09-27 02:39:09 on dist-test-slave-zv8z"
server_key: "1fd58f5117724b5f7982e9ce0d802635"
server_key_iv: "bb9189c87dab42a2928f61d660bf07fe"
server_key_version: "encryptionkey@0"
I20260927 02:39:09.592250 6832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-2/wal/instance:
uuid: "95c9bea2bd4044729172198bd0ee9bd4"
format_stamp: "Formatted at 2026-09-27 02:39:09 on dist-test-slave-zv8z"
server_key: "1fd58f5117724b5f7982e9ce0d802635"
server_key_iv: "bb9189c87dab42a2928f61d660bf07fe"
server_key_version: "encryptionkey@0"
I20260927 02:39:09.596093 6832 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 02:39:09.598480 6847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:09.599730 6832 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:39:09.599829 6832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-2/wal
uuid: "95c9bea2bd4044729172198bd0ee9bd4"
format_stamp: "Formatted at 2026-09-27 02:39:09 on dist-test-slave-zv8z"
server_key: "1fd58f5117724b5f7982e9ce0d802635"
server_key_iv: "bb9189c87dab42a2928f61d660bf07fe"
server_key_version: "encryptionkey@0"
I20260927 02:39:09.599915 6832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:09.614070 6832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:39:09.616842 6832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:39:09.617012 6832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:39:09.617544 6832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:39:09.618330 6832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:39:09.618373 6832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:09.618423 6832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:39:09.618451 6832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:09.626574 6832 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44515
I20260927 02:39:09.626618 6960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44515 every 8 connection(s)
I20260927 02:39:09.627398 6832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:39:09.632158 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 6832
I20260927 02:39:09.632241 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:39:09.632465 392 external_mini_cluster.cc:1428] Setting key 35ffa57b3d58617553a8c3e427aa0c1f
Sep 27 02:39:09 dist-test-slave-zv8z krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476749, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:39:09.639863 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:09.629004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50229 (local address 127.0.98.62:35857)
0927 02:39:09.629138 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:09.629141 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:09.629942 (+ 801us) server_negotiation.cc:407] Connection header received
0927 02:39:09.631234 (+ 1292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:09.631238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:09.631293 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:09.631366 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:09.632515 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:09.633141 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:09.634376 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:09.634652 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:09.634734 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:09.636936 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:09.636939 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:09.636941 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:09.636963 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:09.638132 (+ 1169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:09.638598 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:09.638601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:09.638602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:09.638687 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:09.639024 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:09.639026 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:09.639027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:09.639181 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:09.639247 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:09.639662 (+ 415us) server_negotiation.cc:299] Negotiation successful
0927 02:39:09.639749 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 02:39:09.640590 6963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:09.629381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35857 (local address 127.0.98.3:50229)
0927 02:39:09.629786 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:09.629825 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:09.631036 (+ 1211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:09.631537 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:09.631543 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:09.631864 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:09.632377 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:09.632392 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:09.633285 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:09.633289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:09.634271 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:09.634282 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:09.634786 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:09.634789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:09.634962 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:09.635456 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:09.635471 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:09.636795 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:09.638225 (+ 1430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:09.638230 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:09.638240 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:09.638501 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:09.638789 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:09.638792 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:09.638794 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:09.638927 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:09.639248 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:09.639253 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:09.639472 (+ 219us) client_negotiation.cc:769] Sending connection context
0927 02:39:09.639614 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 02:39:09.639840 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":104,"threads_started":1}
I20260927 02:39:09.641587 6961 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35857
I20260927 02:39:09.641793 6961 heartbeater.cc:464] Registering TS with master...
I20260927 02:39:09.642239 6961 heartbeater.cc:511] Master 127.0.98.62:35857 requested a full tablet report, sending...
I20260927 02:39:09.643164 6502 ts_manager.cc:194] Registered new tserver with Master: 95c9bea2bd4044729172198bd0ee9bd4 (127.0.98.3:44515)
I20260927 02:39:09.643666 6502 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50229
I20260927 02:39:09.646080 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260927 02:39:09.672789 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40345
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35857,127.0.98.61:40345
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:39:09.778395 6966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:39:09.778649 6966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:39:09.778703 6966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:39:09.782670 6966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:39:09.782725 6966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:39:09.782752 6966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:39:09.782773 6966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:39:09.782799 6966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:39:09.787937 6966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35857,127.0.98.61:40345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
W20260927 02:39:09.788128 6966 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:35857,127.0.98.61:40345'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 02:39:09.788916 6966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:39:09.789872 6966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:39:09.797766 6971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:09.797849 6974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:09.797837 6966 server_base.cc:1034] running on GCE node
W20260927 02:39:09.797780 6972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:09.798494 6966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:39:09.799551 6966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:39:09.800699 6966 hybrid_clock.cc:648] HybridClock initialized: now 1790476749800676 us; error 38 us; skew 500 ppm
Sep 27 02:39:09 dist-test-slave-zv8z krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476749, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:39:09.803278 6966 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260927 02:39:09.804136 6966 webserver.cc:460] Webserver started at http://127.0.98.61:39411/ using document root <none> and password file <none>
I20260927 02:39:09.804642 6966 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:09.804704 6966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:09.804877 6966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:39:09.806816 6966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-1/data/instance:
uuid: "6f4dad5a3ce0416eb97e8a4be06dff38"
format_stamp: "Formatted at 2026-09-27 02:39:09 on dist-test-slave-zv8z"
server_key: "1a59dfccf4b698ae948599e253d539e1"
server_key_iv: "17d262835cc78f75754b53dd03f3ac8f"
server_key_version: "encryptionkey@0"
I20260927 02:39:09.807324 6966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-1/wal/instance:
uuid: "6f4dad5a3ce0416eb97e8a4be06dff38"
format_stamp: "Formatted at 2026-09-27 02:39:09 on dist-test-slave-zv8z"
server_key: "1a59dfccf4b698ae948599e253d539e1"
server_key_iv: "17d262835cc78f75754b53dd03f3ac8f"
server_key_version: "encryptionkey@0"
I20260927 02:39:09.810726 6966 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:39:09.812983 6981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:09.814162 6966 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:39:09.814289 6966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-1/wal
uuid: "6f4dad5a3ce0416eb97e8a4be06dff38"
format_stamp: "Formatted at 2026-09-27 02:39:09 on dist-test-slave-zv8z"
server_key: "1a59dfccf4b698ae948599e253d539e1"
server_key_iv: "17d262835cc78f75754b53dd03f3ac8f"
server_key_version: "encryptionkey@0"
I20260927 02:39:09.814381 6966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:09.828825 6966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:39:09.831602 6966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:39:09.831768 6966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:39:09.838266 6966 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40345
I20260927 02:39:09.838531 7033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40345 every 8 connection(s)
I20260927 02:39:09.839184 6966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-1/data/info.pb
I20260927 02:39:09.841787 7034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:39:09.845361 7034 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:39:09.848322 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 6966
Sep 27 02:39:09 dist-test-slave-zv8z krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476749, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 02:39:09 dist-test-slave-zv8z krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 02:39:09.859001 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:09.847252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39342 (local address 127.0.98.62:35857)
0927 02:39:09.847381 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:09.847385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:09.848273 (+ 888us) server_negotiation.cc:407] Connection header received
0927 02:39:09.849381 (+ 1108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:09.849385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:09.849442 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:09.849498 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:09.850847 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:09.851352 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:09.852426 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:09.852643 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:09.852728 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:09.855270 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:09.855274 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:09.855276 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:09.855299 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:09.856749 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:09.857348 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:09.857351 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:09.857352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:09.857448 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:09.857790 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:09.857793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:09.857795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:09.858016 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:09.858107 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:09.858702 (+ 595us) server_negotiation.cc:299] Negotiation successful
0927 02:39:09.858832 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 02:39:09.859836 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:09.847608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35857 (local address 127.0.0.1:39342)
0927 02:39:09.848103 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:09.848159 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:09.849126 (+ 967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:09.849765 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:09.849772 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:09.850209 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:09.850717 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:09.850734 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:09.851470 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:09.851474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:09.852318 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:09.852326 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:09.852770 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:09.852774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:09.852911 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:09.853437 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:09.853457 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:09.855120 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:09.856859 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:09.856866 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:09.856876 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:09.857243 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:09.857552 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:09.857555 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:09.857557 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:09.857701 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:09.858120 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:09.858127 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:09.858437 (+ 310us) client_negotiation.cc:769] Sending connection context
0927 02:39:09.858647 (+ 210us) client_negotiation.cc:240] Negotiation successful
0927 02:39:09.858975 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":43,"spinlock_wait_cycles":1792,"thread_start_us":110,"threads_started":1}
I20260927 02:39:09.859835 7038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:09.849190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34606 (local address 127.0.98.61:40345)
0927 02:39:09.849555 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:09.849566 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:09.849603 (+ 37us) server_negotiation.cc:407] Connection header received
0927 02:39:09.849740 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:09.849745 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:09.849934 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:09.850104 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:09.850747 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:09.851730 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:09.852518 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:09.852869 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:09.852960 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:09.855658 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:09.855663 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:09.855673 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:09.855692 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:09.857313 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:09.857682 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:09.857692 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:09.857693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:09.857769 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:09.858040 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:09.858044 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:09.858046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:09.858420 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:09.858610 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:09.858821 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 02:39:09.859138 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":162,"server-negotiator.queue_time_us":273,"spinlock_wait_cycles":10496,"thread_start_us":100,"threads_started":1}
I20260927 02:39:09.861207 7034 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40345 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:39:09 dist-test-slave-zv8z krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476749, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 02:39:09.868503 7038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:09.861951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34610 (local address 127.0.98.61:40345)
0927 02:39:09.862049 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:09.862052 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:09.862062 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:39:09.862087 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:09.862089 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:09.862120 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:09.862178 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:09.862580 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:09.863064 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:09.863790 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:09.864015 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:09.864087 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:09.865895 (+ 1808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:09.865898 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:09.865900 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:09.865924 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:09.867191 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:09.867581 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:09.867584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:09.867585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:09.867656 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:09.867883 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:09.867886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:09.867887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:09.868043 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:09.868121 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:09.868287 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 02:39:09.868387 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":14,"server-negotiator.queue_time_us":36}
I20260927 02:39:09.868516 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:09.861641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40345 (local address 127.0.0.1:34610)
0927 02:39:09.861738 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:09.861752 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:09.861814 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:09.862224 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:09.862226 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:09.862385 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:09.862476 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:09.862481 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:09.863164 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:09.863167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:09.863701 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:09.863709 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:09.864096 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:09.864098 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:09.864169 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:09.864644 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:09.864655 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:09.865780 (+ 1125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:09.867293 (+ 1513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:09.867295 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:09.867297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:09.867464 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:09.867730 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:09.867732 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:09.867733 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:09.867793 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:09.868114 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:09.868116 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:09.868178 (+ 62us) client_negotiation.cc:769] Sending connection context
0927 02:39:09.868246 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 02:39:09.868338 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 02:39:09.871410 7034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38: Bootstrap starting.
I20260927 02:39:09.873538 7034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38: Neither blocks nor log segments found. Creating new log.
I20260927 02:39:09.874203 7034 log.cc:824] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38: Log is configured to *not* fsync() on all Append() calls
I20260927 02:39:09.876004 7034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38: No bootstrap required, opened a new log
I20260927 02:39:09.879187 7034 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } } peers { permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40345 } }
I20260927 02:39:09.879396 7034 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:39:09.879431 7034 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f4dad5a3ce0416eb97e8a4be06dff38, State: Initialized, Role: FOLLOWER
I20260927 02:39:09.879899 7034 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } } peers { permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40345 } }
I20260927 02:39:09.880309 7040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } } peers { permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40345 } } }
I20260927 02:39:09.880417 7040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:39:09.880968 7034 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 02:39:09.885149 7051 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:39:09.885215 7051 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:39:09.887167 6966 master_runner.cc:428] Detected that this master 6f4dad5a3ce0416eb97e8a4be06dff38 is joining an existing cluster
I20260927 02:39:09.887224 6966 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:40345
I20260927 02:39:09.897857 7038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:09.889332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34618 (local address 127.0.98.61:40345)
0927 02:39:09.889437 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:09.889442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:09.889810 (+ 368us) server_negotiation.cc:407] Connection header received
0927 02:39:09.889912 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:09.889915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:09.889953 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:09.890030 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:09.890621 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:09.891366 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:09.892478 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:09.892727 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:09.892806 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:09.894007 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:09.894011 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:09.894013 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:09.894035 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:09.895752 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:09.896147 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:09.896151 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:09.896153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:09.896224 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:09.897003 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:09.897006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:09.897007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:09.897181 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:09.897271 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:09.897532 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 02:39:09.897656 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":40}
I20260927 02:39:09.897867 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:09.889522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40345 (local address 127.0.0.1:34618)
0927 02:39:09.889730 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:09.889742 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:09.889830 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:09.890125 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:09.890128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:09.890396 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:09.890518 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:09.890534 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:09.891467 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:09.891471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:09.892377 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:09.892387 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:09.892840 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:09.892844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:09.892927 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:09.893425 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:09.893438 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:09.893896 (+ 458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:09.895852 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:09.895854 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:09.895857 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:09.896071 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:09.896834 (+ 763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:09.896836 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:09.896837 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:09.896924 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:09.897267 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:09.897269 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:09.897356 (+ 87us) client_negotiation.cc:769] Sending connection context
0927 02:39:09.897465 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 02:39:09.897588 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"spinlock_wait_cycles":1792,"thread_start_us":76,"threads_started":1}
I20260927 02:39:09.898473 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:09.889333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39358 (local address 127.0.98.62:35857)
0927 02:39:09.889437 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:09.889442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:09.889678 (+ 236us) server_negotiation.cc:407] Connection header received
0927 02:39:09.889869 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:09.889874 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:09.889924 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:09.889994 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:09.890563 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:09.891368 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:09.893537 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:09.893861 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:09.893986 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:09.895213 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:09.895217 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:09.895219 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:09.895246 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:09.896742 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:09.897201 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:09.897205 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:09.897207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:09.897285 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:09.897522 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:09.897526 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:09.897527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:09.897744 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:09.897813 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:09.898186 (+ 373us) server_negotiation.cc:299] Negotiation successful
0927 02:39:09.898333 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 02:39:09.898969 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:09.889221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35857 (local address 127.0.0.1:39358)
0927 02:39:09.889596 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:09.889611 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:09.889744 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:09.890022 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:09.890025 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:09.890323 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:09.890457 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:09.890463 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:09.892509 (+ 2046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:09.892513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:09.893427 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:09.893437 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:09.893969 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:09.893973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:09.894073 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:09.894645 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:09.894659 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:09.895092 (+ 433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:09.896836 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:09.896840 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:09.896842 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:09.897107 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:09.897350 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:09.897353 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:09.897355 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:09.897427 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:09.897907 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:09.897909 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:09.897992 (+ 83us) client_negotiation.cc:769] Sending connection context
0927 02:39:09.898672 (+ 680us) client_negotiation.cc:240] Negotiation successful
0927 02:39:09.898795 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":1,"thread_start_us":81,"threads_started":1}
Sep 27 02:39:09 dist-test-slave-zv8z krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476748, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 02:39:09.911568 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:09.902801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40345 (local address 127.0.0.1:34624)
0927 02:39:09.903157 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:09.903213 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:09.903327 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:09.903706 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:09.903712 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:09.904182 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:09.904302 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:09.904309 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:09.904940 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:09.904944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:09.905923 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:09.905932 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:09.906345 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:09.906349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:09.906457 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:09.906902 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:09.906916 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:09.908509 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:09.910148 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:09.910156 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:09.910162 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:09.910412 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:09.910644 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:09.910646 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:09.910647 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:09.910713 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:09.911077 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:09.911082 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:09.911182 (+ 100us) client_negotiation.cc:769] Sending connection context
0927 02:39:09.911296 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 02:39:09.911437 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":78,"threads_started":1}
I20260927 02:39:09.911593 7038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:09.903401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34624 (local address 127.0.98.61:40345)
0927 02:39:09.903492 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:09.903496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:09.903508 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:39:09.903560 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:09.903562 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:09.903601 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:09.903662 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:09.904399 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:09.904854 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:09.906042 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:09.906236 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:09.906335 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:09.908636 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:09.908640 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:09.908642 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:09.908665 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:09.910045 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:09.910489 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:09.910493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:09.910495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:09.910572 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:09.910777 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:09.910780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:09.910782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:09.910965 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:09.911060 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:09.911350 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 02:39:09.911449 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 02:39:09.913061 6502 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:40345: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 40345
}
attrs {
promote: true
}
}
dest_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f"
cas_config_opid_index: -1
I20260927 02:39:09.913640 6502 consensus_queue.cc:237] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } } peers { permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40345 } attrs { promote: true } }
I20260927 02:39:09.915277 7061 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 6f4dad5a3ce0416eb97e8a4be06dff38 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } } peers { permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40345 } attrs { promote: true } } }
I20260927 02:39:09.915830 7008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:39:09.917038 7008 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [term 1 FOLLOWER]: Refusing update from remote peer 372b9126cbeb4dbdbd3b50c5bfa23a8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 02:39:09.917069 7061 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:40345
I20260927 02:39:09.917532 6966 master.cc:537] Master@127.0.98.61:40345 shutting down...
I20260927 02:39:09.917860 7061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } } peers { permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40345 } attrs { promote: true } } }
I20260927 02:39:09.917946 7061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f [sys.catalog]: This master's current role is: LEADER
I20260927 02:39:09.918123 7061 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40345 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 02:39:09.919788 6489 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260927 02:39:09.920753 6489 consensus_peers.cc:487] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f -> Peer 6f4dad5a3ce0416eb97e8a4be06dff38 (127.0.98.61:40345): Couldn't send request to peer 6f4dad5a3ce0416eb97e8a4be06dff38. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 02:39:09.921516 6966 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:39:09.921767 6966 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:39:09.921862 6966 tablet_replica.cc:330] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38: stopping tablet replica
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
I20260927 02:39:09.935824 6966 master.cc:552] Master@127.0.98.61:40345 shutdown complete.
I20260927 02:39:09.935877 6966 master_runner.cc:305] Clearing existing system tablet
I20260927 02:39:09.936087 6966 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:09.936133 6966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:09.942224 6966 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.002s sys 0.000s
I20260927 02:39:09.943277 7068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:09.943471 6966 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:09.943593 6966 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:39:09.943635 6966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-1/wal
uuid: "6f4dad5a3ce0416eb97e8a4be06dff38"
format_stamp: "Formatted at 2026-09-27 02:39:09 on dist-test-slave-zv8z"
server_key: "1a59dfccf4b698ae948599e253d539e1"
server_key_iv: "17d262835cc78f75754b53dd03f3ac8f"
server_key_version: "encryptionkey@0"
I20260927 02:39:09.945395 6966 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38: Deleting tablet data with delete state TABLET_DATA_DELETED
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260927 02:39:09.947387 6966 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 02:39:09.947472 6966 log.cc:1196] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 02:39:09.947705 6966 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38: Deleting consensus metadata
I20260927 02:39:09.947693 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33141
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35857,127.0.98.61:40345,127.0.98.60:33141
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:39:09.947860 6966 master_runner.cc:315] Copying system tablet from 127.0.98.62:35857
I20260927 02:39:09.948992 6966 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:35857
I20260927 02:39:09.956157 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:09.949648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39372 (local address 127.0.98.62:35857)
0927 02:39:09.949768 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:09.949773 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:09.949976 (+ 203us) server_negotiation.cc:407] Connection header received
0927 02:39:09.950101 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:09.950105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:09.950152 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:09.950218 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:09.950681 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:09.951195 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:09.951973 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:09.952192 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:09.952277 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:09.953321 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:09.953325 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:09.953326 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:09.953349 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:09.954655 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:09.955030 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:09.955033 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:09.955035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:09.955108 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:09.955349 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:09.955352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:09.955354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:09.955534 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:09.955619 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:09.955899 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 02:39:09.956002 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 02:39:09.956172 7075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:09.949578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35857 (local address 127.0.0.1:39372)
0927 02:39:09.949882 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:09.949894 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:09.950006 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:09.950269 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:09.950272 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:09.950463 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:09.950568 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:09.950574 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:09.951295 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:09.951298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:09.951865 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:09.951872 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:09.952290 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:09.952294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:09.952381 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:09.952852 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:09.952864 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:09.953171 (+ 307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:09.954753 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:09.954756 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:09.954759 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:09.954945 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:09.955188 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:09.955191 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:09.955193 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:09.955274 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:09.955620 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:09.955623 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:09.955714 (+ 91us) client_negotiation.cc:769] Sending connection context
0927 02:39:09.955842 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 02:39:09.955980 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":77,"threads_started":1}
I20260927 02:39:09.956866 6522 tablet_copy_service.cc:140] P 372b9126cbeb4dbdbd3b50c5bfa23a8f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 6f4dad5a3ce0416eb97e8a4be06dff38 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:39372)
I20260927 02:39:09.957020 6522 tablet_copy_service.cc:161] P 372b9126cbeb4dbdbd3b50c5bfa23a8f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 6f4dad5a3ce0416eb97e8a4be06dff38 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:39372: session id = 6f4dad5a3ce0416eb97e8a4be06dff38-00000000000000000000000000000000
I20260927 02:39:09.958448 6522 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 02:39:09.959858 6966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 02:39:09.961586 6966 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38: tablet copy: Starting download of 0 data blocks...
I20260927 02:39:09.961758 6966 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38: tablet copy: Starting download of 1 WAL segments...
I20260927 02:39:09.963223 6966 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 02:39:09.964634 6522 tablet_copy_service.cc:342] P 372b9126cbeb4dbdbd3b50c5bfa23a8f: Request end of tablet copy session 6f4dad5a3ce0416eb97e8a4be06dff38-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:39372
I20260927 02:39:09.964728 6522 tablet_copy_service.cc:433] P 372b9126cbeb4dbdbd3b50c5bfa23a8f: ending tablet copy session 6f4dad5a3ce0416eb97e8a4be06dff38-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 6f4dad5a3ce0416eb97e8a4be06dff38
I20260927 02:39:09.970929 6966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:39:09.972788 7079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:09.973069 6966 server_base.cc:1034] running on GCE node
W20260927 02:39:09.979188 7076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:09.982380 7077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:09.982626 6966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:39:09.982911 6966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:39:09.984041 6966 hybrid_clock.cc:648] HybridClock initialized: now 1790476749984032 us; error 32 us; skew 500 ppm
Sep 27 02:39:09 dist-test-slave-zv8z krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476749, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:39:09.985623 6966 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260927 02:39:09.986054 6966 webserver.cc:460] Webserver started at http://127.0.98.61:39685/ using document root <none> and password file <none>
I20260927 02:39:09.986232 6966 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:09.986307 6966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:09.987551 6966 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:39:09.988368 7086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:09.988574 6966 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:39:09.988639 6966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-1/wal
uuid: "6f4dad5a3ce0416eb97e8a4be06dff38"
format_stamp: "Formatted at 2026-09-27 02:39:09 on dist-test-slave-zv8z"
server_key: "1a59dfccf4b698ae948599e253d539e1"
server_key_iv: "17d262835cc78f75754b53dd03f3ac8f"
server_key_version: "encryptionkey@0"
I20260927 02:39:09.988711 6966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:10.000188 6966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:39:10.002024 6966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:39:10.006567 6966 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40345
I20260927 02:39:10.006599 7138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40345 every 8 connection(s)
I20260927 02:39:10.007129 6966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-1/data/info.pb
I20260927 02:39:10.008497 7139 sys_catalog.cc:263] Verifying existing consensus state
I20260927 02:39:10.009269 7139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38: Bootstrap starting.
I20260927 02:39:10.017033 7139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:39:10.017295 7139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38: Bootstrap complete.
I20260927 02:39:10.017791 7139 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } } peers { permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40345 } attrs { promote: true } }
I20260927 02:39:10.017910 7139 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6f4dad5a3ce0416eb97e8a4be06dff38, State: Initialized, Role: LEARNER
I20260927 02:39:10.018007 7139 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } } peers { permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40345 } attrs { promote: true } }
I20260927 02:39:10.018321 7142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } } peers { permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40345 } attrs { promote: true } } }
I20260927 02:39:10.018455 7139 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:39:10.018483 7142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [sys.catalog]: This master's current role is: LEARNER
I20260927 02:39:10.021672 7153 catalog_manager.cc:1259] Loaded cluster ID: 7432986bc1b74b229d125db051eba32d
I20260927 02:39:10.021720 7153 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38: loading cluster ID for follower catalog manager: success
I20260927 02:39:10.023237 7153 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38: acquiring CA information for follower catalog manager: success
I20260927 02:39:10.024276 7153 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 02:39:10.088235 7073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:39:10.088595 7073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:39:10.088660 7073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:39:10.092782 7073 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:39:10.092840 7073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:39:10.092859 7073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:39:10.092875 7073 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:39:10.092892 7073 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:39:10.098407 7073 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35857,127.0.98.61:40345,127.0.98.60:33141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:39:10.099534 7073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:39:10.100610 7073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:39:10.108831 7159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:10.108894 7158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:10.108839 7161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:10.108937 7073 server_base.cc:1034] running on GCE node
I20260927 02:39:10.110421 7073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:39:10.111456 7073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:39:10.112605 7073 hybrid_clock.cc:648] HybridClock initialized: now 1790476750112588 us; error 31 us; skew 500 ppm
Sep 27 02:39:10 dist-test-slave-zv8z krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476750, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:39:10.115207 7073 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260927 02:39:10.116007 7073 webserver.cc:460] Webserver started at http://127.0.98.60:39489/ using document root <none> and password file <none>
I20260927 02:39:10.116577 7073 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:10.116655 7073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:10.116840 7073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:39:10.119247 7073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-2/data/instance:
uuid: "50ce6beeca7d4e7186a77b7c34dcb011"
format_stamp: "Formatted at 2026-09-27 02:39:10 on dist-test-slave-zv8z"
server_key: "5502b4c4c7384e7043e9b08bdd83dc48"
server_key_iv: "b34df731a994701b3e812f86a7ccd4e0"
server_key_version: "encryptionkey@0"
I20260927 02:39:10.119796 7073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-2/wal/instance:
uuid: "50ce6beeca7d4e7186a77b7c34dcb011"
format_stamp: "Formatted at 2026-09-27 02:39:10 on dist-test-slave-zv8z"
server_key: "5502b4c4c7384e7043e9b08bdd83dc48"
server_key_iv: "b34df731a994701b3e812f86a7ccd4e0"
server_key_version: "encryptionkey@0"
I20260927 02:39:10.123310 7073 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:39:10.125640 7168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:10.126920 7073 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:39:10.127048 7073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-2/wal
uuid: "50ce6beeca7d4e7186a77b7c34dcb011"
format_stamp: "Formatted at 2026-09-27 02:39:10 on dist-test-slave-zv8z"
server_key: "5502b4c4c7384e7043e9b08bdd83dc48"
server_key_iv: "b34df731a994701b3e812f86a7ccd4e0"
server_key_version: "encryptionkey@0"
I20260927 02:39:10.127137 7073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:10.152421 7073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:39:10.155346 7073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:39:10.155511 7073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:39:10.162765 7073 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:33141
I20260927 02:39:10.163061 7220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:33141 every 8 connection(s)
I20260927 02:39:10.163693 7073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-2/data/info.pb
I20260927 02:39:10.166774 7221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:39:10.170130 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 7073
I20260927 02:39:10.170348 7221 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:39:10 dist-test-slave-zv8z krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476750, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 02:39:10 dist-test-slave-zv8z krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 02:39:10.182319 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:10.172179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39380 (local address 127.0.98.62:35857)
0927 02:39:10.172312 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:10.172316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:10.172523 (+ 207us) server_negotiation.cc:407] Connection header received
0927 02:39:10.173073 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:10.173077 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:10.173129 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:10.173200 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:10.174101 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.174650 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.175776 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.176015 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.176093 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:10.178630 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:10.178635 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:10.178637 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:10.178660 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:10.180137 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:10.180661 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:10.180665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:10.180666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:10.180747 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:10.181071 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:10.181074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:10.181076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:10.181253 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:10.181340 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:10.182020 (+ 680us) server_negotiation.cc:299] Negotiation successful
0927 02:39:10.182120 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 02:39:10.182947 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:10.172033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35857 (local address 127.0.0.1:39380)
0927 02:39:10.172401 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:10.172444 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:10.172880 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:10.173272 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:10.173277 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:10.173684 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:10.173971 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:10.173983 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.174761 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.174764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:10.175651 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:10.175660 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.176149 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.176153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:10.176361 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:10.176953 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:10.176970 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:10.178467 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:10.180242 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:10.180251 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:10.180260 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:10.180573 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:10.180827 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:10.180830 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:10.180832 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:10.180994 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:10.181345 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:10.181351 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:10.181580 (+ 229us) client_negotiation.cc:769] Sending connection context
0927 02:39:10.181737 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 02:39:10.182026 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":51,"spinlock_wait_cycles":3968,"thread_start_us":122,"threads_started":1}
I20260927 02:39:10.182951 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:10.171645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53030 (local address 127.0.98.60:33141)
0927 02:39:10.172028 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:10.172046 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:10.172085 (+ 39us) server_negotiation.cc:407] Connection header received
0927 02:39:10.172875 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:10.172893 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:10.173081 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:10.173362 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:10.173912 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.174899 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.175817 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.176200 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.176334 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:10.179006 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:10.179011 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:10.179039 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:10.179064 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:10.180585 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:10.180982 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:10.180987 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:10.180988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:10.181066 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:10.181298 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:10.181302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:10.181304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:10.181574 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:10.181708 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:10.181971 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 02:39:10.182110 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":251,"thread_start_us":109,"threads_started":1}
I20260927 02:39:10.184206 7221 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40345 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:39:10 dist-test-slave-zv8z krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476750, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 02:39:10.191797 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:10.184695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40345 (local address 127.0.0.1:34640)
0927 02:39:10.184804 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:10.184817 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:10.184883 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:10.185242 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:10.185244 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:10.185368 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:10.185462 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:10.185467 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.186341 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.186344 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:10.186888 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:10.186895 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.187375 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.187379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:10.187463 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:10.187959 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:10.187970 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:10.189118 (+ 1148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:10.190693 (+ 1575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:10.190696 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:10.190697 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:10.190871 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:10.191089 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:10.191091 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:10.191092 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:10.191157 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:10.191442 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:10.191444 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:10.191514 (+ 70us) client_negotiation.cc:769] Sending connection context
0927 02:39:10.191591 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 02:39:10.191683 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 02:39:10.191850 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:10.184795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34640 (local address 127.0.98.61:40345)
0927 02:39:10.185040 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:10.185044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:10.185058 (+ 14us) server_negotiation.cc:407] Connection header received
0927 02:39:10.185104 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:10.185106 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:10.185145 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:10.185215 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:10.185564 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.186220 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.187001 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.187223 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.187305 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:10.189241 (+ 1936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:10.189244 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:10.189246 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:10.189268 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:10.190601 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:10.190951 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:10.190954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:10.190955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:10.191018 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:10.191219 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:10.191221 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:10.191222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:10.191378 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:10.191448 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:10.191631 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 02:39:10.191733 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":88,"threads_started":1}
I20260927 02:39:10.192554 7221 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33141 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:39:10 dist-test-slave-zv8z krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476750, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 02:39:10.199393 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:10.192991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33141 (local address 127.0.0.1:53042)
0927 02:39:10.193098 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:10.193111 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:10.193191 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:10.193369 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:10.193371 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:10.193516 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:10.193620 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:10.193625 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.194370 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.194373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:10.194880 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:10.194887 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.195294 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.195297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:10.195405 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:10.195768 (+ 363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:10.195777 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:10.196829 (+ 1052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:10.198120 (+ 1291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:10.198123 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:10.198125 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:10.198322 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:10.198551 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:10.198554 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:10.198556 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:10.198626 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:10.198926 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:10.198929 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:10.199004 (+ 75us) client_negotiation.cc:769] Sending connection context
0927 02:39:10.199085 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 02:39:10.199210 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":17}
I20260927 02:39:10.199393 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:10.193041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53042 (local address 127.0.98.60:33141)
0927 02:39:10.193124 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:10.193128 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:10.193155 (+ 27us) server_negotiation.cc:407] Connection header received
0927 02:39:10.193258 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:10.193261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:10.193296 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:10.193351 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:10.193710 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.194283 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.194971 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.195209 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.195288 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:10.196961 (+ 1673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:10.196965 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:10.196966 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:10.196985 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:10.198031 (+ 1046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:10.198396 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:10.198399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:10.198401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:10.198464 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:10.198694 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:10.198697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:10.198699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:10.198847 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:10.198934 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:10.199132 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 02:39:10.199254 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":33}
I20260927 02:39:10.202629 7221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011: Bootstrap starting.
I20260927 02:39:10.204909 7221 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011: Neither blocks nor log segments found. Creating new log.
I20260927 02:39:10.205582 7221 log.cc:824] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011: Log is configured to *not* fsync() on all Append() calls
I20260927 02:39:10.207394 7221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011: No bootstrap required, opened a new log
I20260927 02:39:10.210178 7221 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } } peers { permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40345 } } peers { permanent_uuid: "50ce6beeca7d4e7186a77b7c34dcb011" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33141 } }
I20260927 02:39:10.210423 7221 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:39:10.210477 7221 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50ce6beeca7d4e7186a77b7c34dcb011, State: Initialized, Role: FOLLOWER
I20260927 02:39:10.210899 7221 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } } peers { permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40345 } } peers { permanent_uuid: "50ce6beeca7d4e7186a77b7c34dcb011" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33141 } }
I20260927 02:39:10.211326 7227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } } peers { permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40345 } } peers { permanent_uuid: "50ce6beeca7d4e7186a77b7c34dcb011" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33141 } } }
I20260927 02:39:10.211447 7227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:39:10.211984 7221 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 02:39:10.216073 7238 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:39:10.216143 7238 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:39:10.219532 7073 master_runner.cc:428] Detected that this master 50ce6beeca7d4e7186a77b7c34dcb011 is joining an existing cluster
I20260927 02:39:10.219587 7073 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:33141
I20260927 02:39:10.236253 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 6564
I20260927 02:39:10.241773 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:35281
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35857
--webserver_port=37125
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:35857,127.0.98.61:40345
--tserver_master_addrs=127.0.98.62:35857,127.0.98.61:40345,127.0.98.60:33141 with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:39:10.250976 7245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:10.222034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33141 (local address 127.0.0.1:53048)
0927 02:39:10.222297 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:10.222310 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:10.222397 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:10.222578 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:10.222580 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:10.222835 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:10.222939 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:10.222944 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.223809 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.223812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:10.224636 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:10.224644 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.225252 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.225255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:10.225398 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:10.226384 (+ 986us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:10.226399 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:10.247302 (+ 20903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:10.249499 (+ 2197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:10.249502 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:10.249504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:10.249722 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:10.249956 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:10.249958 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:10.249960 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:10.250033 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:10.250448 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:10.250451 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:10.250539 (+ 88us) client_negotiation.cc:769] Sending connection context
0927 02:39:10.250641 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 02:39:10.250766 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":29,"thread_start_us":72,"threads_started":1}
I20260927 02:39:10.251462 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:10.221937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53048 (local address 127.0.98.60:33141)
0927 02:39:10.222025 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:10.222029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:10.222366 (+ 337us) server_negotiation.cc:407] Connection header received
0927 02:39:10.222459 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:10.222462 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:10.222497 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:10.222558 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:10.223037 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.223717 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.224742 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.225059 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.225124 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:10.247443 (+ 22319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:10.247446 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:10.247449 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:10.247473 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:10.249391 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:10.249806 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:10.249809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:10.249810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:10.249880 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:10.250120 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:10.250123 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:10.250125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:10.250332 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:10.250417 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:10.251217 (+ 800us) server_negotiation.cc:299] Negotiation successful
0927 02:39:10.251323 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28,"spinlock_wait_cycles":768}
I20260927 02:39:10.253870 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:10.221496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35857 (local address 127.0.0.1:39382)
0927 02:39:10.221765 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:10.221780 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:10.221870 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:10.223170 (+ 1300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:10.223173 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:10.223375 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:10.223476 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:10.223482 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.224471 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.224474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:10.225368 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:10.225376 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.226314 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.226317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:10.226393 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:10.226773 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:10.226785 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:10.246849 (+ 20064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:10.252389 (+ 5540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:10.252392 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:10.252394 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:10.252600 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:10.252892 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:10.252895 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:10.252896 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:10.252974 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:10.253399 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:10.253402 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:10.253497 (+ 95us) client_negotiation.cc:769] Sending connection context
0927 02:39:10.253586 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 02:39:10.253707 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":34,"thread_start_us":114,"threads_started":1}
I20260927 02:39:10.254324 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:10.221540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39382 (local address 127.0.98.62:35857)
0927 02:39:10.222956 (+ 1416us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:10.222960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:10.222975 (+ 15us) server_negotiation.cc:407] Connection header received
0927 02:39:10.223014 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:10.223017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:10.223072 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:10.223159 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:10.223599 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.224359 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.225480 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.226092 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.226176 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:10.247847 (+ 21671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:10.247850 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:10.247853 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:10.247887 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:10.252267 (+ 4380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:10.252709 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:10.252713 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:10.252714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:10.252798 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:10.253077 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:10.253080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:10.253082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:10.253274 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:10.253357 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:10.254033 (+ 676us) server_negotiation.cc:299] Negotiation successful
0927 02:39:10.254159 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1334}
I20260927 02:39:10.258371 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:10.225531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40345 (local address 127.0.0.1:34646)
0927 02:39:10.225748 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:10.225761 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:10.225845 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:10.226176 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:10.226179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:10.246558 (+ 20379us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:10.246739 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:10.246750 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.248951 (+ 2201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.248956 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:10.249849 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:10.249857 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.250533 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.250537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:10.250637 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:10.251184 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:10.251197 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:10.251558 (+ 361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:10.256849 (+ 5291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:10.256852 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:10.256854 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:10.257076 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:10.257359 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:10.257362 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:10.257364 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:10.257447 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:10.257890 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:10.257893 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:10.257978 (+ 85us) client_negotiation.cc:769] Sending connection context
0927 02:39:10.258068 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 02:39:10.258191 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":85,"threads_started":1}
I20260927 02:39:10.259748 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:10.225420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34646 (local address 127.0.98.61:40345)
0927 02:39:10.225521 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:10.225536 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:10.225820 (+ 284us) server_negotiation.cc:407] Connection header received
0927 02:39:10.225952 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:10.225956 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:10.226004 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:10.226051 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:10.247013 (+ 20962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.248373 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.249970 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.250333 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.250420 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:10.254771 (+ 4351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:10.254774 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:10.254777 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:10.254804 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:10.256738 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:10.257180 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:10.257183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:10.257185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:10.257270 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:10.257542 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:10.257545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:10.257546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:10.257758 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:10.257849 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:10.259481 (+ 1632us) server_negotiation.cc:299] Negotiation successful
0927 02:39:10.259603 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
Sep 27 02:39:10 dist-test-slave-zv8z krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476748, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 02:39:10.265136 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:10.258027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53058 (local address 127.0.98.60:33141)
0927 02:39:10.258145 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:10.258149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:10.258190 (+ 41us) server_negotiation.cc:407] Connection header received
0927 02:39:10.258345 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:10.258348 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:10.258400 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:10.258482 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:10.258878 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.259342 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.260549 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.260763 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.260850 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:10.262614 (+ 1764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:10.262618 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:10.262620 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:10.262639 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:10.263793 (+ 1154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:10.264154 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:10.264158 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:10.264159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:10.264223 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:10.264450 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:10.264453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:10.264455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:10.264607 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:10.264686 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:10.264913 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 02:39:10.265002 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 02:39:10.265148 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:10.257795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33141 (local address 127.0.0.1:53058)
0927 02:39:10.258034 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:10.258047 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:10.258120 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:10.258503 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:10.258506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:10.258665 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:10.258767 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:10.258774 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.259454 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.259457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:10.260190 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:10.260200 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.260867 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.260871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:10.260961 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:10.261331 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:10.261344 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:10.262477 (+ 1133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:10.263890 (+ 1413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:10.263893 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:10.263896 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:10.264073 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:10.264300 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:10.264303 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:10.264305 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:10.264377 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:10.264691 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:10.264694 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:10.264778 (+ 84us) client_negotiation.cc:769] Sending connection context
0927 02:39:10.264858 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 02:39:10.264986 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169}
I20260927 02:39:10.266214 6502 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:33141: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "50ce6beeca7d4e7186a77b7c34dcb011"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 33141
}
attrs {
promote: true
}
}
dest_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f"
cas_config_opid_index: 5
I20260927 02:39:10.266788 6502 consensus_queue.cc:237] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } } peers { permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40345 } attrs { promote: true } } peers { permanent_uuid: "50ce6beeca7d4e7186a77b7c34dcb011" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33141 } attrs { promote: true } }
I20260927 02:39:10.267725 7062 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 50ce6beeca7d4e7186a77b7c34dcb011 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } } peers { permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40345 } attrs { promote: true } } peers { permanent_uuid: "50ce6beeca7d4e7186a77b7c34dcb011" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33141 } attrs { promote: true } } }
I20260927 02:39:10.268241 7195 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:39:10.269163 7062 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:33141
I20260927 02:39:10.269484 7195 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011 [term 1 FOLLOWER]: Refusing update from remote peer 372b9126cbeb4dbdbd3b50c5bfa23a8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 02:39:10.269479 7063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } } peers { permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40345 } attrs { promote: true } } peers { permanent_uuid: "50ce6beeca7d4e7186a77b7c34dcb011" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33141 } attrs { promote: true } } }
I20260927 02:39:10.269584 7063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f [sys.catalog]: This master's current role is: LEADER
I20260927 02:39:10.270159 7063 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "50ce6beeca7d4e7186a77b7c34dcb011" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33141 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:39:10.270270 7073 master.cc:537] Master@127.0.98.60:33141 shutting down...
W20260927 02:39:10.271798 6487 consensus_peers.cc:487] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f -> Peer 50ce6beeca7d4e7186a77b7c34dcb011 (127.0.98.60:33141): Couldn't send request to peer 50ce6beeca7d4e7186a77b7c34dcb011. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 02:39:10.275079 7073 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:39:10.275344 7073 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:39:10.275453 7073 tablet_replica.cc:330] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011: stopping tablet replica
I20260927 02:39:10.277158 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:10.267989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40345 (local address 127.0.0.1:34658)
0927 02:39:10.269929 (+ 1940us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:10.269944 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:10.270009 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:10.273037 (+ 3028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:10.273040 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:10.273058 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:39:10.273139 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:10.273145 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.274142 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.274146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:10.275532 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:10.275543 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.276761 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.276765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:10.276852 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:10.276897 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 02:39:10.276966 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 02:39:10.277033 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1871}
I20260927 02:39:10.277844 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:10.269398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34658 (local address 127.0.98.61:40345)
0927 02:39:10.272312 (+ 2914us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:10.272317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:10.272331 (+ 14us) server_negotiation.cc:407] Connection header received
0927 02:39:10.272374 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:10.272377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:10.272431 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:10.272903 (+ 472us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:39:10.273255 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.274014 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.275660 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.276374 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.276561 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:10.276701 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:10.277651 (+ 950us) server_negotiation.cc:299] Negotiation successful
0927 02:39:10.277726 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2831}
I20260927 02:39:10.278402 7113 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [term 1 LEARNER]: Refusing update from remote peer 372b9126cbeb4dbdbd3b50c5bfa23a8f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 02:39:10.278913 7063 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40345 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 02:39:10.281100 7073 master.cc:552] Master@127.0.98.60:33141 shutdown complete.
I20260927 02:39:10.280825 7113 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 50ce6beeca7d4e7186a77b7c34dcb011 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } } peers { permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40345 } attrs { promote: true } } peers { permanent_uuid: "50ce6beeca7d4e7186a77b7c34dcb011" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33141 } attrs { promote: true } } }
I20260927 02:39:10.281141 7073 master_runner.cc:305] Clearing existing system tablet
I20260927 02:39:10.281353 7073 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:10.281391 7073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:10.282773 7073 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:39:10.283185 7142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 372b9126cbeb4dbdbd3b50c5bfa23a8f. Latest consensus state: current_term: 1 leader_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } } peers { permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40345 } attrs { promote: true } } peers { permanent_uuid: "50ce6beeca7d4e7186a77b7c34dcb011" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33141 } attrs { promote: true } } }
I20260927 02:39:10.283301 7142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [sys.catalog]: This master's current role is: LEARNER
I20260927 02:39:10.283439 7142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } } peers { permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40345 } attrs { promote: true } } peers { permanent_uuid: "50ce6beeca7d4e7186a77b7c34dcb011" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33141 } attrs { promote: true } } }
I20260927 02:39:10.283521 7142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [sys.catalog]: This master's current role is: LEARNER
I20260927 02:39:10.283805 7251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:10.284008 7073 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:10.284116 7073 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:39:10.284147 7073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-2/wal
uuid: "50ce6beeca7d4e7186a77b7c34dcb011"
format_stamp: "Formatted at 2026-09-27 02:39:10 on dist-test-slave-zv8z"
server_key: "5502b4c4c7384e7043e9b08bdd83dc48"
server_key_iv: "b34df731a994701b3e812f86a7ccd4e0"
server_key_version: "encryptionkey@0"
I20260927 02:39:10.285754 7073 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:39:10.286602 7063 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f: attempting to promote NON_VOTER 6f4dad5a3ce0416eb97e8a4be06dff38 to VOTER
I20260927 02:39:10.287060 7063 consensus_queue.cc:237] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } } peers { permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40345 } attrs { promote: false } } peers { permanent_uuid: "50ce6beeca7d4e7186a77b7c34dcb011" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33141 } attrs { promote: true } }
I20260927 02:39:10.288322 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:10.288063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33141 (local address 127.0.0.1:53066)
0927 02:39:10.288180 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:10.288249 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 02:39:10.288753 7073 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 02:39:10.288823 7073 log.cc:1196] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 02:39:10.288856 7113 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [term 1 LEARNER]: Refusing update from remote peer 372b9126cbeb4dbdbd3b50c5bfa23a8f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 02:39:10.289052 7073 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011: Deleting consensus metadata
I20260927 02:39:10.289345 7061 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40345 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 02:39:10.289790 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:10.289494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33141 (local address 127.0.0.1:53078)
0927 02:39:10.289677 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:10.289726 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
W20260927 02:39:10.289963 6487 consensus_peers.cc:487] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f -> Peer 50ce6beeca7d4e7186a77b7c34dcb011 (127.0.98.60:33141): Couldn't send request to peer 50ce6beeca7d4e7186a77b7c34dcb011. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33141: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 02:39:10.290330 7073 master_runner.cc:315] Copying system tablet from 127.0.98.62:35857
I20260927 02:39:10.290874 7062 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 6f4dad5a3ce0416eb97e8a4be06dff38 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } } peers { permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40345 } attrs { promote: false } } peers { permanent_uuid: "50ce6beeca7d4e7186a77b7c34dcb011" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33141 } attrs { promote: true } } }
I20260927 02:39:10.291306 7113 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 6f4dad5a3ce0416eb97e8a4be06dff38 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } } peers { permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40345 } attrs { promote: false } } peers { permanent_uuid: "50ce6beeca7d4e7186a77b7c34dcb011" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33141 } attrs { promote: true } } }
I20260927 02:39:10.292177 7062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } } peers { permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40345 } attrs { promote: false } } peers { permanent_uuid: "50ce6beeca7d4e7186a77b7c34dcb011" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33141 } attrs { promote: true } } }
I20260927 02:39:10.292266 7062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f [sys.catalog]: This master's current role is: LEADER
I20260927 02:39:10.292387 7062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } } peers { permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40345 } attrs { promote: false } } peers { permanent_uuid: "50ce6beeca7d4e7186a77b7c34dcb011" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33141 } attrs { promote: true } } }
I20260927 02:39:10.292457 7062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f [sys.catalog]: This master's current role is: LEADER
I20260927 02:39:10.292968 7250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } } peers { permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40345 } attrs { promote: false } } peers { permanent_uuid: "50ce6beeca7d4e7186a77b7c34dcb011" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33141 } attrs { promote: true } } }
I20260927 02:39:10.293130 7250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:39:10.293718 7073 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:35857
I20260927 02:39:10.304661 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:10.294199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39398 (local address 127.0.98.62:35857)
0927 02:39:10.294359 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:10.294363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:10.298190 (+ 3827us) server_negotiation.cc:407] Connection header received
0927 02:39:10.298317 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:10.298319 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:10.298363 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:10.298432 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:10.298902 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.299506 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.300283 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.300495 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.300575 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:10.301622 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:10.301625 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:10.301628 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:10.301652 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:10.303133 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:10.303504 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:10.303507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:10.303509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:10.303585 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:10.303880 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:10.303883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:10.303884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:10.304063 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:10.304144 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:10.304409 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 02:39:10.304504 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260927 02:39:10.304674 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:10.294155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35857 (local address 127.0.0.1:39398)
0927 02:39:10.298088 (+ 3933us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:10.298100 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:10.298217 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:10.298460 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:10.298462 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:10.298702 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:10.298795 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:10.298801 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.299604 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.299607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:10.300184 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:10.300191 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.300594 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.300598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:10.300676 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:10.301182 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:10.301195 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:10.301502 (+ 307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:10.303228 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:10.303231 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:10.303233 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:10.303419 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:10.303671 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:10.303674 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:10.303676 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:10.303801 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:10.304144 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:10.304147 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:10.304236 (+ 89us) client_negotiation.cc:769] Sending connection context
0927 02:39:10.304322 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 02:39:10.304460 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3866,"thread_start_us":97,"threads_started":1}
I20260927 02:39:10.305606 6522 tablet_copy_service.cc:140] P 372b9126cbeb4dbdbd3b50c5bfa23a8f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 50ce6beeca7d4e7186a77b7c34dcb011 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:39398)
I20260927 02:39:10.305696 6522 tablet_copy_service.cc:161] P 372b9126cbeb4dbdbd3b50c5bfa23a8f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 50ce6beeca7d4e7186a77b7c34dcb011 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:39398: session id = 50ce6beeca7d4e7186a77b7c34dcb011-00000000000000000000000000000000
I20260927 02:39:10.306573 6522 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 02:39:10.307655 7073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 02:39:10.309577 7073 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011: tablet copy: Starting download of 0 data blocks...
I20260927 02:39:10.309749 7073 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011: tablet copy: Starting download of 1 WAL segments...
I20260927 02:39:10.311059 7073 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 02:39:10.312597 6522 tablet_copy_service.cc:342] P 372b9126cbeb4dbdbd3b50c5bfa23a8f: Request end of tablet copy session 50ce6beeca7d4e7186a77b7c34dcb011-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:39398
I20260927 02:39:10.312669 6522 tablet_copy_service.cc:433] P 372b9126cbeb4dbdbd3b50c5bfa23a8f: ending tablet copy session 50ce6beeca7d4e7186a77b7c34dcb011-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 50ce6beeca7d4e7186a77b7c34dcb011
I20260927 02:39:10.319276 7073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:39:10.320894 7258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:10.321357 7261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:10.322410 7073 server_base.cc:1034] running on GCE node
W20260927 02:39:10.326829 7259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:10.327112 7073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:39:10.327405 7073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:39:10.328532 7073 hybrid_clock.cc:648] HybridClock initialized: now 1790476750328523 us; error 30 us; skew 500 ppm
Sep 27 02:39:10 dist-test-slave-zv8z krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476750, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:39:10.329875 7073 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260927 02:39:10.330312 7073 webserver.cc:460] Webserver started at http://127.0.98.60:46673/ using document root <none> and password file <none>
I20260927 02:39:10.330488 7073 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:10.330523 7073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:10.331542 7073 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:39:10.332351 7268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:10.332556 7073 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:39:10.332635 7073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-2/wal
uuid: "50ce6beeca7d4e7186a77b7c34dcb011"
format_stamp: "Formatted at 2026-09-27 02:39:10 on dist-test-slave-zv8z"
server_key: "5502b4c4c7384e7043e9b08bdd83dc48"
server_key_iv: "b34df731a994701b3e812f86a7ccd4e0"
server_key_version: "encryptionkey@0"
I20260927 02:39:10.332717 7073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:10.344734 7073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:39:10.346645 7073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:39:10.351158 7073 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:33141
I20260927 02:39:10.351171 7320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:33141 every 8 connection(s)
I20260927 02:39:10.351691 7073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/master-2/data/info.pb
I20260927 02:39:10.353065 7321 sys_catalog.cc:263] Verifying existing consensus state
I20260927 02:39:10.353869 7321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011: Bootstrap starting.
I20260927 02:39:10.362407 7321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:39:10.362666 7321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011: Bootstrap complete.
I20260927 02:39:10.363148 7321 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } } peers { permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40345 } attrs { promote: false } } peers { permanent_uuid: "50ce6beeca7d4e7186a77b7c34dcb011" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33141 } attrs { promote: true } }
I20260927 02:39:10.363261 7321 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 50ce6beeca7d4e7186a77b7c34dcb011, State: Initialized, Role: LEARNER
I20260927 02:39:10.363356 7321 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } } peers { permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40345 } attrs { promote: false } } peers { permanent_uuid: "50ce6beeca7d4e7186a77b7c34dcb011" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33141 } attrs { promote: true } }
I20260927 02:39:10.363777 7321 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:39:10.363716 7324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } } peers { permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40345 } attrs { promote: false } } peers { permanent_uuid: "50ce6beeca7d4e7186a77b7c34dcb011" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33141 } attrs { promote: true } } }
I20260927 02:39:10.363950 7324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011 [sys.catalog]: This master's current role is: LEARNER
I20260927 02:39:10.367249 7335 catalog_manager.cc:1259] Loaded cluster ID: 7432986bc1b74b229d125db051eba32d
I20260927 02:39:10.367293 7335 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011: loading cluster ID for follower catalog manager: success
I20260927 02:39:10.368744 7335 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011: acquiring CA information for follower catalog manager: success
I20260927 02:39:10.369848 7335 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 02:39:10.398983 7247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:39:10.399355 7247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:39:10.399448 7247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:39:10.405230 7247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:39:10.405313 7247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:39:10.405396 7247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:39:10.411123 7247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:35281
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37125
--tserver_master_addrs=127.0.98.62:35857,127.0.98.61:40345,127.0.98.60:33141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:39:10.412097 7247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:39:10.413028 7247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:39:10.422148 7343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:10.422127 7341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:10.422432 7340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:10.424809 7247 server_base.cc:1034] running on GCE node
I20260927 02:39:10.425189 7247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:39:10.425788 7247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:39:10.426940 7247 hybrid_clock.cc:648] HybridClock initialized: now 1790476750426914 us; error 45 us; skew 500 ppm
Sep 27 02:39:10 dist-test-slave-zv8z krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476750, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:39:10.429935 7247 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 02:39:10.430768 7247 webserver.cc:460] Webserver started at http://127.0.98.1:37125/ using document root <none> and password file <none>
I20260927 02:39:10.431290 7247 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:10.431363 7247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:10.435168 7247 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260927 02:39:10.437170 7350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:10.438388 7247 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:39:10.438481 7247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-0/wal
uuid: "6c73821393684ab0961970fdda44e818"
format_stamp: "Formatted at 2026-09-27 02:39:09 on dist-test-slave-zv8z"
server_key: "b5eb7b8b2e1d5497368435e62c6be8f8"
server_key_iv: "b179f0e41004e42dbebe3dbb02b16509"
server_key_version: "encryptionkey@0"
I20260927 02:39:10.438979 7247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:10.442723 6827 heartbeater.cc:503] Master 127.0.98.62:35857 was elected leader, sending a full tablet report...
I20260927 02:39:10.452478 7247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:39:10.455420 7247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:39:10.455578 7247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:39:10.456121 7247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:39:10.456895 7247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:39:10.456938 7247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:10.456979 7247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:39:10.457007 7247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:10.465518 7247 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35281
I20260927 02:39:10.465566 7463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35281 every 8 connection(s)
I20260927 02:39:10.466462 7247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:39:10.470408 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 7247
I20260927 02:39:10.470595 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 6698
Sep 27 02:39:10 dist-test-slave-zv8z krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476750, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 02:39:10 dist-test-slave-zv8z krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476750, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 02:39:10.478508 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38069
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35857
--webserver_port=41865
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:35857,127.0.98.61:40345
--tserver_master_addrs=127.0.98.62:35857,127.0.98.61:40345,127.0.98.60:33141 with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 02:39:10 dist-test-slave-zv8z krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476750, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 02:39:10.482214 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:10.469562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52815 (local address 127.0.98.62:35857)
0927 02:39:10.469679 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:10.469682 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:10.469814 (+ 132us) server_negotiation.cc:407] Connection header received
0927 02:39:10.470605 (+ 791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:10.470609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:10.470654 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:10.470717 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:10.471908 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.472677 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.474552 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.474872 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.474942 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:10.477961 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:10.477965 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:10.477968 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:10.477995 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:10.479819 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:10.480598 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:10.480602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:10.480604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:10.480693 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:10.481238 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:10.481241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:10.481243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:10.481448 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:10.481518 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:10.481926 (+ 408us) server_negotiation.cc:299] Negotiation successful
0927 02:39:10.482039 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 02:39:10.482905 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:10.468061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42779 (local address 127.0.98.60:33141)
0927 02:39:10.468390 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:10.468394 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:10.469343 (+ 949us) server_negotiation.cc:407] Connection header received
0927 02:39:10.470339 (+ 996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:10.470342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:10.470393 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:10.470484 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:10.471908 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.472920 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.474548 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.474896 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.474977 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:10.478285 (+ 3308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:10.478290 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:10.478293 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:10.478319 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:10.479914 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:10.480535 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:10.480539 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:10.480541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:10.480628 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:10.480984 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:10.480987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:10.480988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:10.481200 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:10.481284 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:10.482621 (+ 1337us) server_negotiation.cc:299] Negotiation successful
0927 02:39:10.482744 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":82,"threads_started":1}
I20260927 02:39:10.483206 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:10.468134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40507 (local address 127.0.98.61:40345)
0927 02:39:10.468258 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:10.468262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:10.469135 (+ 873us) server_negotiation.cc:407] Connection header received
0927 02:39:10.470115 (+ 980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:10.470119 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:10.470178 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:10.470280 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:10.471909 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.472709 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.474550 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.474892 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.474965 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:10.479832 (+ 4867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:10.479836 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:10.479839 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:10.479865 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:10.481565 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:10.481993 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:10.481996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:10.481998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:10.482075 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:10.482383 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:10.482386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:10.482388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:10.482583 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:10.482670 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:10.482966 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 02:39:10.483073 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 02:39:10.483479 7471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:10.469466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35857 (local address 127.0.98.1:52815)
0927 02:39:10.469733 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:10.469745 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:10.469923 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:10.470814 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:10.470817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:10.471039 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:10.471762 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:10.471777 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.472832 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.472835 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:10.474430 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:10.474440 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.475121 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.475125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:10.475378 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:10.476057 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:10.476089 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:10.477768 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:10.479925 (+ 2157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:10.479931 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:10.479946 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:10.480483 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:10.481023 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:10.481026 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:10.481028 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:10.481127 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:10.481586 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:10.481589 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:10.481677 (+ 88us) client_negotiation.cc:769] Sending connection context
0927 02:39:10.482383 (+ 706us) client_negotiation.cc:240] Negotiation successful
0927 02:39:10.482512 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":320,"spinlock_wait_cycles":33024,"thread_start_us":75,"threads_started":1}
I20260927 02:39:10.483939 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:10.468504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40345 (local address 127.0.98.1:40507)
0927 02:39:10.468971 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:10.469021 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:10.469914 (+ 893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:10.470440 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:10.470446 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:10.471377 (+ 931us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:10.471754 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:10.471773 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.472837 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.472840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:10.474433 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:10.474440 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.475121 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.475125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:10.475401 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:10.476185 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:10.476198 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:10.479676 (+ 3478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:10.481676 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:10.481679 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:10.481681 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:10.481898 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:10.482177 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:10.482180 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:10.482181 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:10.482285 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:10.482722 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:10.482725 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:10.482804 (+ 79us) client_negotiation.cc:769] Sending connection context
0927 02:39:10.483383 (+ 579us) client_negotiation.cc:240] Negotiation successful
0927 02:39:10.483511 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":382,"spinlock_wait_cycles":6272,"thread_start_us":514,"threads_started":1}
I20260927 02:39:10.484663 7464 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35857
I20260927 02:39:10.484855 7464 heartbeater.cc:464] Registering TS with master...
I20260927 02:39:10.484967 7470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:10.468504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33141 (local address 127.0.98.1:42779)
0927 02:39:10.469264 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:10.469276 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:10.469923 (+ 647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:10.470519 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:10.470522 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:10.471088 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:10.471754 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:10.471773 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.473029 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.473032 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:10.474424 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:10.474434 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.475121 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.475125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:10.475397 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:10.476086 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:10.476115 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:10.478127 (+ 2012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:10.480087 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:10.480090 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:10.480093 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:10.480429 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:10.480719 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:10.480722 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:10.480725 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:10.480884 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:10.481335 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:10.481341 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:10.481620 (+ 279us) client_negotiation.cc:769] Sending connection context
0927 02:39:10.481829 (+ 209us) client_negotiation.cc:240] Negotiation successful
0927 02:39:10.482159 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"mutex_wait_us":300,"spinlock_wait_cycles":10880,"thread_start_us":104,"threads_started":1}
I20260927 02:39:10.485275 7464 heartbeater.cc:511] Master 127.0.98.62:35857 requested a full tablet report, sending...
I20260927 02:39:10.485725 7466 heartbeater.cc:347] Connected to a master server at 127.0.98.61:40345
I20260927 02:39:10.485801 7466 heartbeater.cc:464] Registering TS with master...
I20260927 02:39:10.485948 7465 heartbeater.cc:347] Connected to a master server at 127.0.98.60:33141
I20260927 02:39:10.485956 7466 heartbeater.cc:511] Master 127.0.98.61:40345 requested a full tablet report, sending...
I20260927 02:39:10.486022 7465 heartbeater.cc:464] Registering TS with master...
I20260927 02:39:10.486140 7465 heartbeater.cc:511] Master 127.0.98.60:33141 requested a full tablet report, sending...
I20260927 02:39:10.486456 6502 ts_manager.cc:194] Re-registered known tserver with Master: 6c73821393684ab0961970fdda44e818 (127.0.98.1:35281)
I20260927 02:39:10.487008 7095 ts_manager.cc:194] Registered new tserver with Master: 6c73821393684ab0961970fdda44e818 (127.0.98.1:35281)
I20260927 02:39:10.487008 7277 ts_manager.cc:194] Registered new tserver with Master: 6c73821393684ab0961970fdda44e818 (127.0.98.1:35281)
I20260927 02:39:10.487186 6502 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52815
W20260927 02:39:10.586970 7472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:39:10.587216 7472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:39:10.587272 7472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:39:10.591393 7472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:39:10.591455 7472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:39:10.591529 7472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:39:10.596705 7472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38069
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=41865
--tserver_master_addrs=127.0.98.62:35857,127.0.98.61:40345,127.0.98.60:33141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:39:10.597657 7472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:39:10.598616 7472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:39:10.607666 7478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:10.607709 7472 server_base.cc:1034] running on GCE node
W20260927 02:39:10.607628 7480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:10.607658 7477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:10.608208 7472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:39:10.608762 7472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:39:10.609912 7472 hybrid_clock.cc:648] HybridClock initialized: now 1790476750609893 us; error 28 us; skew 500 ppm
Sep 27 02:39:10 dist-test-slave-zv8z krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476750, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:39:10.612453 7472 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 02:39:10.613219 7472 webserver.cc:460] Webserver started at http://127.0.98.2:41865/ using document root <none> and password file <none>
I20260927 02:39:10.613689 7472 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:10.613754 7472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:10.617275 7472 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:39:10.619252 7487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:10.620461 7472 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:39:10.620550 7472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-1/wal
uuid: "bfb91636c48348d1abc15549cd284922"
format_stamp: "Formatted at 2026-09-27 02:39:09 on dist-test-slave-zv8z"
server_key: "46878191608bcfef1679238110e2ca4f"
server_key_iv: "1a7ca4173c58b856141f965d276b204e"
server_key_version: "encryptionkey@0"
I20260927 02:39:10.621030 7472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:10.635636 7472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:39:10.638587 7472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:39:10.638731 7472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:39:10.639233 7472 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:39:10.640038 7472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:39:10.640084 7472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:10.640141 7472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:39:10.640168 7472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:10.645588 6961 heartbeater.cc:503] Master 127.0.98.62:35857 was elected leader, sending a full tablet report...
I20260927 02:39:10.649739 7472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38069
I20260927 02:39:10.650858 7600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38069 every 8 connection(s)
I20260927 02:39:10.650906 7472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:39:10.660116 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 7472
I20260927 02:39:10.660290 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 6832
Sep 27 02:39:10 dist-test-slave-zv8z krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476750, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 02:39:10 dist-test-slave-zv8z krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476750, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 02:39:10.667315 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44515
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35857
--webserver_port=37539
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:35857,127.0.98.61:40345
--tserver_master_addrs=127.0.98.62:35857,127.0.98.61:40345,127.0.98.60:33141 with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 02:39:10 dist-test-slave-zv8z krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476750, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:39:10.669611 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:10.652625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48269 (local address 127.0.98.60:33141)
0927 02:39:10.652878 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:10.652882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:10.654918 (+ 2036us) server_negotiation.cc:407] Connection header received
0927 02:39:10.655564 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:10.655568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:10.655617 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:10.655665 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:10.657679 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.658647 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.661308 (+ 2661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.661652 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.661755 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:10.664988 (+ 3233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:10.664992 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:10.664995 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:10.665025 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:10.667080 (+ 2055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:10.667742 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:10.667747 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:10.667749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:10.667838 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:10.668263 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:10.668266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:10.668269 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:10.668483 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:10.668554 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:10.669306 (+ 752us) server_negotiation.cc:299] Negotiation successful
0927 02:39:10.669443 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180}
I20260927 02:39:10.671329 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:10.652617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42769 (local address 127.0.98.61:40345)
0927 02:39:10.652754 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:10.652758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:10.654323 (+ 1565us) server_negotiation.cc:407] Connection header received
0927 02:39:10.655555 (+ 1232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:10.655558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:10.655613 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:10.655665 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:10.657958 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.658724 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.663206 (+ 4482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.663536 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.663617 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:10.666045 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:10.666048 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:10.666051 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:10.666079 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:10.669403 (+ 3324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:10.669967 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:10.669971 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:10.669973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:10.670075 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:10.670406 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:10.670410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:10.670412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:10.670629 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:10.670698 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:10.671058 (+ 360us) server_negotiation.cc:299] Negotiation successful
0927 02:39:10.671167 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 02:39:10.671911 7606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:10.653054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33141 (local address 127.0.98.2:48269)
0927 02:39:10.654829 (+ 1775us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:10.654842 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:10.655306 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:10.656059 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:10.656067 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:10.656591 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:10.657483 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:10.657518 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.658825 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.658828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:10.660405 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:10.660415 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.661810 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.661814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:10.662001 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:10.662678 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:10.662723 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:10.664806 (+ 2083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:10.667204 (+ 2398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:10.667210 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:10.667227 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:10.667633 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:10.667940 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:10.667943 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:10.667945 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:10.668118 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:10.668687 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:10.668694 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:10.669023 (+ 329us) client_negotiation.cc:769] Sending connection context
0927 02:39:10.669971 (+ 948us) client_negotiation.cc:240] Negotiation successful
0927 02:39:10.670352 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1689,"mutex_wait_us":179,"spinlock_wait_cycles":37760,"thread_start_us":134,"threads_started":1}
I20260927 02:39:10.672410 7605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:10.653054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40345 (local address 127.0.98.2:42769)
0927 02:39:10.654150 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:10.654205 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:10.655306 (+ 1101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:10.656059 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:10.656068 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:10.657721 (+ 1653us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:10.657820 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:10.657826 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.659045 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.659048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:10.663067 (+ 4019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:10.663076 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.663774 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.663777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:10.663880 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:10.664387 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:10.664402 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:10.665875 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:10.669517 (+ 3642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:10.669521 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:10.669523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:10.669857 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:10.670189 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:10.670192 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:10.670194 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:10.670304 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:10.670769 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:10.670772 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:10.670861 (+ 89us) client_negotiation.cc:769] Sending connection context
0927 02:39:10.671673 (+ 812us) client_negotiation.cc:240] Negotiation successful
0927 02:39:10.671882 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":884,"mutex_wait_us":77,"thread_start_us":655,"threads_started":1}
I20260927 02:39:10.672742 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:10.654477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47285 (local address 127.0.98.62:35857)
0927 02:39:10.654585 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:10.654590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:10.656172 (+ 1582us) server_negotiation.cc:407] Connection header received
0927 02:39:10.656376 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:10.656379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:10.656446 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:10.656517 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:10.657631 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.658452 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.666191 (+ 7739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.666571 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.666683 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:10.668942 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:10.668946 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:10.668949 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:10.668976 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:10.670760 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:10.671344 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:10.671347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:10.671349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:10.671437 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:10.671775 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:10.671780 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:10.671782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:10.671991 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:10.672062 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:10.672481 (+ 419us) server_negotiation.cc:299] Negotiation successful
0927 02:39:10.672608 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 02:39:10.673194 7607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:10.654155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35857 (local address 127.0.98.2:47285)
0927 02:39:10.656084 (+ 1929us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:10.656097 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:10.656241 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:10.657190 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:10.657193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:10.657380 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:10.657495 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:10.657504 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.658858 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.658861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:10.662842 (+ 3981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:10.662852 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.666857 (+ 4005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.666863 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:10.666948 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:10.667324 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:10.667339 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:10.668780 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:10.670855 (+ 2075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:10.670858 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:10.670999 (+ 141us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:10.671238 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:10.671559 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:10.671562 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:10.671564 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:10.671659 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:10.672126 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:10.672129 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:10.672216 (+ 87us) client_negotiation.cc:769] Sending connection context
0927 02:39:10.672913 (+ 697us) client_negotiation.cc:240] Negotiation successful
0927 02:39:10.673046 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1862,"mutex_wait_us":135,"thread_start_us":100,"threads_started":1}
I20260927 02:39:10.673763 7603 heartbeater.cc:347] Connected to a master server at 127.0.98.61:40345
I20260927 02:39:10.674018 7603 heartbeater.cc:464] Registering TS with master...
I20260927 02:39:10.674633 7603 heartbeater.cc:511] Master 127.0.98.61:40345 requested a full tablet report, sending...
I20260927 02:39:10.675036 7601 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35857
I20260927 02:39:10.675117 7601 heartbeater.cc:464] Registering TS with master...
I20260927 02:39:10.675283 7601 heartbeater.cc:511] Master 127.0.98.62:35857 requested a full tablet report, sending...
I20260927 02:39:10.675383 7602 heartbeater.cc:347] Connected to a master server at 127.0.98.60:33141
I20260927 02:39:10.675449 7602 heartbeater.cc:464] Registering TS with master...
I20260927 02:39:10.675601 7602 heartbeater.cc:511] Master 127.0.98.60:33141 requested a full tablet report, sending...
I20260927 02:39:10.676066 6502 ts_manager.cc:194] Re-registered known tserver with Master: bfb91636c48348d1abc15549cd284922 (127.0.98.2:38069)
I20260927 02:39:10.676255 7277 ts_manager.cc:194] Registered new tserver with Master: bfb91636c48348d1abc15549cd284922 (127.0.98.2:38069)
I20260927 02:39:10.676792 6502 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47285
I20260927 02:39:10.677017 7095 ts_manager.cc:194] Registered new tserver with Master: bfb91636c48348d1abc15549cd284922 (127.0.98.2:38069)
I20260927 02:39:10.745543 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:10.742407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33141 (local address 127.0.0.1:47774)
0927 02:39:10.742518 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:10.742535 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:10.742628 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:10.742874 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:10.742877 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:10.742895 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:39:10.742982 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:10.742987 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.743754 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.743757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:10.744498 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:10.744506 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.745144 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.745148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:10.745232 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:10.745254 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 02:39:10.745327 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 02:39:10.745400 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 02:39:10.745543 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:10.742484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47774 (local address 127.0.98.60:33141)
0927 02:39:10.742650 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:10.742654 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:10.742685 (+ 31us) server_negotiation.cc:407] Connection header received
0927 02:39:10.742731 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:10.742733 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:10.742780 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:10.742850 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:39:10.743073 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.743652 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.744617 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.745028 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.745121 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:10.745264 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:10.745340 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 02:39:10.745404 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 02:39:10.746205 7295 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260927 02:39:10.746631 7324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 372b9126cbeb4dbdbd3b50c5bfa23a8f. Latest consensus state: current_term: 1 leader_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } } peers { permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40345 } attrs { promote: false } } peers { permanent_uuid: "50ce6beeca7d4e7186a77b7c34dcb011" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33141 } attrs { promote: true } } }
I20260927 02:39:10.746770 7324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011 [sys.catalog]: This master's current role is: LEARNER
W20260927 02:39:10.780980 7608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:39:10.781265 7608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:39:10.781327 7608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:39:10.785240 7608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:39:10.785291 7608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:39:10.785367 7608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:39:10.790484 7608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44515
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=37539
--tserver_master_addrs=127.0.98.62:35857,127.0.98.61:40345,127.0.98.60:33141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:39:10.791414 7608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:39:10.792364 7608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:39:10.801545 7614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:10.801558 7616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:10.801540 7613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:10.801837 7608 server_base.cc:1034] running on GCE node
I20260927 02:39:10.802188 7608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:39:10.802767 7608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:39:10.803938 7608 hybrid_clock.cc:648] HybridClock initialized: now 1790476750803911 us; error 36 us; skew 500 ppm
Sep 27 02:39:10 dist-test-slave-zv8z krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476750, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:39:10.806493 7608 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 02:39:10.807327 7608 webserver.cc:460] Webserver started at http://127.0.98.3:37539/ using document root <none> and password file <none>
I20260927 02:39:10.807826 7608 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:10.807899 7608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:10.811410 7608 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:39:10.813289 7623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:10.814465 7608 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:39:10.814570 7608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-2/wal
uuid: "95c9bea2bd4044729172198bd0ee9bd4"
format_stamp: "Formatted at 2026-09-27 02:39:09 on dist-test-slave-zv8z"
server_key: "1fd58f5117724b5f7982e9ce0d802635"
server_key_iv: "bb9189c87dab42a2928f61d660bf07fe"
server_key_version: "encryptionkey@0"
I20260927 02:39:10.815086 7608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:10.831209 7608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:39:10.834435 7608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:39:10.834599 7608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:39:10.835158 7608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:39:10.835974 7608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:39:10.836020 7608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:10.836071 7608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:39:10.836097 7608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:10.844266 7608 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44515
I20260927 02:39:10.844300 7736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44515 every 8 connection(s)
I20260927 02:39:10.845233 7608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:39:10.853814 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 7608
Sep 27 02:39:10 dist-test-slave-zv8z krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476750, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 27 02:39:10 dist-test-slave-zv8z krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476750, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 02:39:10 dist-test-slave-zv8z krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476750, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 02:39:10.861829 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:10.846984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57729 (local address 127.0.98.60:33141)
0927 02:39:10.847106 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:10.847110 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:10.847985 (+ 875us) server_negotiation.cc:407] Connection header received
0927 02:39:10.849203 (+ 1218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:10.849207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:10.849264 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:10.849335 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:10.850715 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.851523 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.853832 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.854134 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.854208 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:10.858172 (+ 3964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:10.858175 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:10.858178 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:10.858206 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:10.859811 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:10.860220 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:10.860223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:10.860225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:10.860293 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:10.860549 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:10.860552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:10.860553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:10.860736 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:10.860809 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:10.861575 (+ 766us) server_negotiation.cc:299] Negotiation successful
0927 02:39:10.861686 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 02:39:10.862244 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:10.846943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50683 (local address 127.0.98.61:40345)
0927 02:39:10.847076 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:10.847080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:10.848209 (+ 1129us) server_negotiation.cc:407] Connection header received
0927 02:39:10.849230 (+ 1021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:10.849234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:10.849283 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:10.849332 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:10.850753 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.851523 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.852855 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.853062 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.853172 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:10.858647 (+ 5475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:10.858650 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:10.858653 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:10.858678 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:10.860445 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:10.860883 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:10.860886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:10.860888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:10.860963 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:10.861214 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:10.861217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:10.861219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:10.861409 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:10.861486 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:10.861999 (+ 513us) server_negotiation.cc:299] Negotiation successful
0927 02:39:10.862102 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 02:39:10.862709 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:10.847347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33141 (local address 127.0.98.3:57729)
0927 02:39:10.847802 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:10.847852 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:10.849015 (+ 1163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:10.849523 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:10.849529 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:10.849984 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:10.850581 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:10.850594 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.852879 (+ 2285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.852882 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:10.853717 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:10.853726 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.854388 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.854392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:10.854476 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:10.854899 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:10.854909 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:10.858054 (+ 3145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:10.859922 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:10.859925 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:10.859927 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:10.860126 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:10.860381 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:10.860384 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:10.860386 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:10.860459 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:10.860843 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:10.860846 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:10.861104 (+ 258us) client_negotiation.cc:769] Sending connection context
0927 02:39:10.861319 (+ 215us) client_negotiation.cc:240] Negotiation successful
0927 02:39:10.862078 (+ 759us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":565,"spinlock_wait_cycles":31872,"thread_start_us":119,"threads_started":1}
I20260927 02:39:10.862726 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:10.847347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40345 (local address 127.0.98.3:50683)
0927 02:39:10.848126 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:10.848138 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:10.849007 (+ 869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:10.849525 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:10.849532 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:10.849981 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:10.850591 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:10.850598 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.851671 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.851674 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:10.852743 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:10.852752 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.853198 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.853202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:10.853381 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:10.853982 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:10.854002 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:10.858544 (+ 4542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:10.860541 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:10.860544 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:10.860546 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:10.860779 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:10.861053 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:10.861056 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:10.861058 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:10.861131 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:10.861527 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:10.861530 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:10.861622 (+ 92us) client_negotiation.cc:769] Sending connection context
0927 02:39:10.861729 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 02:39:10.861852 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"mutex_wait_us":89,"spinlock_wait_cycles":30464,"thread_start_us":523,"threads_started":1}
I20260927 02:39:10.863018 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:10.848326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35857 (local address 127.0.98.3:36349)
0927 02:39:10.848585 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:10.848599 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:10.849019 (+ 420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:10.849525 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:10.849532 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:10.850007 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:10.850589 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:10.850598 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.852246 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.852250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:10.853068 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:10.853076 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.853794 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.853798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:10.853877 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:10.854415 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:10.854429 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:10.857378 (+ 2949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:10.859575 (+ 2197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:10.859581 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:10.859596 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:10.859908 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:10.860192 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:10.860195 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:10.860196 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:10.860353 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:10.860781 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:10.860788 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:10.861104 (+ 316us) client_negotiation.cc:769] Sending connection context
0927 02:39:10.861319 (+ 215us) client_negotiation.cc:240] Negotiation successful
0927 02:39:10.861662 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":146,"spinlock_wait_cycles":20608,"thread_start_us":81,"threads_started":1}
I20260927 02:39:10.864282 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:10.848631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36349 (local address 127.0.98.62:35857)
0927 02:39:10.848743 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:10.848746 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:10.848758 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:39:10.849230 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:10.849233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:10.849283 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:10.849332 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:10.851652 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.852127 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.853217 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.853577 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.853649 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:10.857655 (+ 4006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:10.857659 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:10.857662 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:10.857685 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:10.859440 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:10.860003 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:10.860007 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:10.860008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:10.860084 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:10.860445 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:10.860448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:10.860450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:10.860648 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:10.860731 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:10.864015 (+ 3284us) server_negotiation.cc:299] Negotiation successful
0927 02:39:10.864137 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 02:39:10.864456 7739 heartbeater.cc:347] Connected to a master server at 127.0.98.61:40345
I20260927 02:39:10.864692 7739 heartbeater.cc:464] Registering TS with master...
I20260927 02:39:10.865240 7739 heartbeater.cc:511] Master 127.0.98.61:40345 requested a full tablet report, sending...
I20260927 02:39:10.865856 7738 heartbeater.cc:347] Connected to a master server at 127.0.98.60:33141
I20260927 02:39:10.865938 7738 heartbeater.cc:464] Registering TS with master...
I20260927 02:39:10.866076 7738 heartbeater.cc:511] Master 127.0.98.60:33141 requested a full tablet report, sending...
I20260927 02:39:10.865864 7737 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35857
I20260927 02:39:10.866328 7737 heartbeater.cc:464] Registering TS with master...
I20260927 02:39:10.866436 7095 ts_manager.cc:194] Registered new tserver with Master: 95c9bea2bd4044729172198bd0ee9bd4 (127.0.98.3:44515)
I20260927 02:39:10.866472 7737 heartbeater.cc:511] Master 127.0.98.62:35857 requested a full tablet report, sending...
I20260927 02:39:10.866982 6502 ts_manager.cc:194] Re-registered known tserver with Master: 95c9bea2bd4044729172198bd0ee9bd4 (127.0.98.3:44515)
I20260927 02:39:10.866974 7277 ts_manager.cc:194] Registered new tserver with Master: 95c9bea2bd4044729172198bd0ee9bd4 (127.0.98.3:44515)
I20260927 02:39:10.867672 6502 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36349
I20260927 02:39:10.870743 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:10.855913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52952 (local address 127.0.98.61:40345)
0927 02:39:10.856105 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:10.856115 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:10.857235 (+ 1120us) server_negotiation.cc:407] Connection header received
0927 02:39:10.857364 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:10.857367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:10.857411 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:10.857479 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:10.863047 (+ 5568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.863933 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.865274 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.865576 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.865669 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:10.867163 (+ 1494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:10.867167 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:10.867170 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:10.867193 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:10.868795 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:10.869292 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:10.869295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:10.869297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:10.869378 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:10.869660 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:10.869663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:10.869665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:10.869870 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:10.869983 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:10.870464 (+ 481us) server_negotiation.cc:299] Negotiation successful
0927 02:39:10.870579 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":69,"threads_started":1}
I20260927 02:39:10.871236 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:10.856435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48472 (local address 127.0.98.62:35857)
0927 02:39:10.856672 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:10.856675 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:10.857499 (+ 824us) server_negotiation.cc:407] Connection header received
0927 02:39:10.857646 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:10.857649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:10.857696 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:10.857780 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:10.858527 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.861216 (+ 2689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.862345 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.862679 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.862753 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:10.864731 (+ 1978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:10.864735 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:10.864737 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:10.864760 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:10.868877 (+ 4117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:10.869779 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:10.869783 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:10.869785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:10.869868 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:10.870158 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:10.870162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:10.870163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:10.870388 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:10.870509 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:10.870942 (+ 433us) server_negotiation.cc:299] Negotiation successful
0927 02:39:10.871065 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":73,"threads_started":1}
I20260927 02:39:10.873471 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:10.857043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47778 (local address 127.0.98.60:33141)
0927 02:39:10.859020 (+ 1977us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:10.859032 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:10.859043 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:39:10.859073 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:10.859076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:10.859117 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:10.859994 (+ 877us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:10.860091 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.867630 (+ 7539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.868717 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.869027 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.869423 (+ 396us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:10.870536 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:10.870539 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:10.870541 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:10.870559 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:10.872061 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:10.872431 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:10.872433 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:10.872434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:10.872501 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:10.872749 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:10.872752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:10.872754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:10.872908 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:10.873009 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:10.873221 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 02:39:10.873324 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1910,"thread_start_us":72,"threads_started":1}
I20260927 02:39:10.874069 6502 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48472:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:39:10.877348 6502 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:39:10.879767 7061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } } peers { permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40345 } attrs { promote: false } } peers { permanent_uuid: "50ce6beeca7d4e7186a77b7c34dcb011" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33141 } attrs { promote: true } } }
I20260927 02:39:10.879904 7061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f [sys.catalog]: This master's current role is: LEADER
I20260927 02:39:10.882882 7755 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f: attempting to promote NON_VOTER 50ce6beeca7d4e7186a77b7c34dcb011 to VOTER
I20260927 02:39:10.883141 7755 consensus_queue.cc:237] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } } peers { permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40345 } attrs { promote: false } } peers { permanent_uuid: "50ce6beeca7d4e7186a77b7c34dcb011" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33141 } attrs { promote: false } }
I20260927 02:39:10.884433 7113 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [term 1 FOLLOWER]: Refusing update from remote peer 372b9126cbeb4dbdbd3b50c5bfa23a8f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 02:39:10.884801 7061 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40345 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 02:39:10.884960 7294 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011 [term 1 LEARNER]: Refusing update from remote peer 372b9126cbeb4dbdbd3b50c5bfa23a8f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 02:39:10.885439 7061 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "50ce6beeca7d4e7186a77b7c34dcb011" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 02:39:10.886353 7756 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 50ce6beeca7d4e7186a77b7c34dcb011 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } } peers { permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40345 } attrs { promote: false } } peers { permanent_uuid: "50ce6beeca7d4e7186a77b7c34dcb011" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33141 } attrs { promote: false } } }
I20260927 02:39:10.886555 7113 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 50ce6beeca7d4e7186a77b7c34dcb011 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } } peers { permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40345 } attrs { promote: false } } peers { permanent_uuid: "50ce6beeca7d4e7186a77b7c34dcb011" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33141 } attrs { promote: false } } }
I20260927 02:39:10.887813 7756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } } peers { permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40345 } attrs { promote: false } } peers { permanent_uuid: "50ce6beeca7d4e7186a77b7c34dcb011" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33141 } attrs { promote: false } } }
I20260927 02:39:10.887914 7756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f [sys.catalog]: This master's current role is: LEADER
I20260927 02:39:10.888036 7061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } } peers { permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40345 } attrs { promote: false } } peers { permanent_uuid: "50ce6beeca7d4e7186a77b7c34dcb011" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33141 } attrs { promote: false } } }
I20260927 02:39:10.888123 7061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f [sys.catalog]: This master's current role is: LEADER
I20260927 02:39:10.887871 7294 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 50ce6beeca7d4e7186a77b7c34dcb011 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } } peers { permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40345 } attrs { promote: false } } peers { permanent_uuid: "50ce6beeca7d4e7186a77b7c34dcb011" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33141 } attrs { promote: false } } }
I20260927 02:39:10.888501 7061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } } peers { permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40345 } attrs { promote: false } } peers { permanent_uuid: "50ce6beeca7d4e7186a77b7c34dcb011" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33141 } attrs { promote: false } } }
I20260927 02:39:10.888586 7061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f [sys.catalog]: This master's current role is: LEADER
I20260927 02:39:10.889071 7764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } } peers { permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40345 } attrs { promote: false } } peers { permanent_uuid: "50ce6beeca7d4e7186a77b7c34dcb011" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33141 } attrs { promote: false } } }
I20260927 02:39:10.889315 7764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:39:10.889879 7324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } } peers { permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40345 } attrs { promote: false } } peers { permanent_uuid: "50ce6beeca7d4e7186a77b7c34dcb011" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33141 } attrs { promote: false } } }
I20260927 02:39:10.889985 7324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:39:10.897009 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:10.891198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35281 (local address 127.0.0.1:39420)
0927 02:39:10.891351 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:10.891365 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:10.891441 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:10.893061 (+ 1620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:10.893065 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:10.893086 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:39:10.893181 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:10.893186 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.894591 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.894595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:10.895519 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:10.895529 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.896607 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.896611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:10.896698 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:10.896724 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 02:39:10.896788 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 02:39:10.896870 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260927 02:39:10.897275 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:10.891730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38069 (local address 127.0.0.1:48376)
0927 02:39:10.892383 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:10.892392 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:10.892447 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:10.893722 (+ 1275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:10.893725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:10.893742 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:39:10.893829 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:10.893836 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.895207 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.895210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:10.896091 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:10.896099 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.896928 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.896931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:10.897021 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:10.897044 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 02:39:10.897092 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 02:39:10.897158 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":90,"threads_started":1}
I20260927 02:39:10.897557 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:10.891361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39420 (local address 127.0.98.1:35281)
0927 02:39:10.892642 (+ 1281us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:10.892661 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:10.892680 (+ 19us) server_negotiation.cc:407] Connection header received
0927 02:39:10.892742 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:10.892747 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:10.892916 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:10.893035 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:39:10.893294 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.894498 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.895656 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.896480 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.896592 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:10.896752 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:10.896816 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 02:39:10.896911 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1191,"thread_start_us":120,"threads_started":1}
I20260927 02:39:10.897855 7768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:10.891878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48376 (local address 127.0.98.2:38069)
0927 02:39:10.893276 (+ 1398us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:10.893286 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:10.893309 (+ 23us) server_negotiation.cc:407] Connection header received
0927 02:39:10.893380 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:10.893384 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:10.893587 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:10.893959 (+ 372us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:39:10.894058 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.895107 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.896196 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.896811 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.897437 (+ 626us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:10.897577 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:10.897625 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 02:39:10.897712 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1304,"thread_start_us":98,"threads_started":1}
I20260927 02:39:10.900050 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:10.891688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44515 (local address 127.0.0.1:49070)
0927 02:39:10.892127 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:10.892139 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:10.892225 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:10.894932 (+ 2707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:10.894935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:10.894950 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:39:10.895031 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:10.895036 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.896356 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.896360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:10.898653 (+ 2293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:10.898664 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.899676 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.899680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:10.899767 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:10.899790 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 02:39:10.899864 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 02:39:10.899928 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":86,"threads_started":1}
I20260927 02:39:10.900279 7769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:10.892059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49070 (local address 127.0.98.3:44515)
0927 02:39:10.894460 (+ 2401us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:10.894470 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:10.894492 (+ 22us) server_negotiation.cc:407] Connection header received
0927 02:39:10.894556 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:10.894561 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:10.894794 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:10.895164 (+ 370us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:39:10.895290 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.896228 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.898787 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:10.899553 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:10.899660 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:10.899791 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:10.899891 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 02:39:10.899979 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2301,"thread_start_us":109,"threads_started":1}
I20260927 02:39:10.900404 7517 tablet_service.cc:1467] Processing CreateTablet for tablet 75ab554f1fda4a61ace553381f5db0c4 (DEFAULT_TABLE table=test-table [id=5a7ca89183b14bf98135799ffdb4e450]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:39:10.900408 7398 tablet_service.cc:1467] Processing CreateTablet for tablet 75ab554f1fda4a61ace553381f5db0c4 (DEFAULT_TABLE table=test-table [id=5a7ca89183b14bf98135799ffdb4e450]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:39:10.901391 7517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75ab554f1fda4a61ace553381f5db0c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:39:10.901422 7398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75ab554f1fda4a61ace553381f5db0c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:39:10.902962 7671 tablet_service.cc:1467] Processing CreateTablet for tablet 75ab554f1fda4a61ace553381f5db0c4 (DEFAULT_TABLE table=test-table [id=5a7ca89183b14bf98135799ffdb4e450]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:39:10.903898 7671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75ab554f1fda4a61ace553381f5db0c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:39:10.907866 7770 tablet_bootstrap.cc:492] T 75ab554f1fda4a61ace553381f5db0c4 P bfb91636c48348d1abc15549cd284922: Bootstrap starting.
I20260927 02:39:10.909672 7771 tablet_bootstrap.cc:492] T 75ab554f1fda4a61ace553381f5db0c4 P 6c73821393684ab0961970fdda44e818: Bootstrap starting.
I20260927 02:39:10.909979 7770 tablet_bootstrap.cc:654] T 75ab554f1fda4a61ace553381f5db0c4 P bfb91636c48348d1abc15549cd284922: Neither blocks nor log segments found. Creating new log.
I20260927 02:39:10.910694 7770 log.cc:824] T 75ab554f1fda4a61ace553381f5db0c4 P bfb91636c48348d1abc15549cd284922: Log is configured to *not* fsync() on all Append() calls
I20260927 02:39:10.912020 7771 tablet_bootstrap.cc:654] T 75ab554f1fda4a61ace553381f5db0c4 P 6c73821393684ab0961970fdda44e818: Neither blocks nor log segments found. Creating new log.
I20260927 02:39:10.912313 7772 tablet_bootstrap.cc:492] T 75ab554f1fda4a61ace553381f5db0c4 P 95c9bea2bd4044729172198bd0ee9bd4: Bootstrap starting.
I20260927 02:39:10.912814 7771 log.cc:824] T 75ab554f1fda4a61ace553381f5db0c4 P 6c73821393684ab0961970fdda44e818: Log is configured to *not* fsync() on all Append() calls
I20260927 02:39:10.913875 7770 tablet_bootstrap.cc:492] T 75ab554f1fda4a61ace553381f5db0c4 P bfb91636c48348d1abc15549cd284922: No bootstrap required, opened a new log
I20260927 02:39:10.914064 7770 ts_tablet_manager.cc:1397] T 75ab554f1fda4a61ace553381f5db0c4 P bfb91636c48348d1abc15549cd284922: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 02:39:10.914395 7771 tablet_bootstrap.cc:492] T 75ab554f1fda4a61ace553381f5db0c4 P 6c73821393684ab0961970fdda44e818: No bootstrap required, opened a new log
I20260927 02:39:10.914433 7772 tablet_bootstrap.cc:654] T 75ab554f1fda4a61ace553381f5db0c4 P 95c9bea2bd4044729172198bd0ee9bd4: Neither blocks nor log segments found. Creating new log.
I20260927 02:39:10.914573 7771 ts_tablet_manager.cc:1397] T 75ab554f1fda4a61ace553381f5db0c4 P 6c73821393684ab0961970fdda44e818: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 02:39:10.915201 7772 log.cc:824] T 75ab554f1fda4a61ace553381f5db0c4 P 95c9bea2bd4044729172198bd0ee9bd4: Log is configured to *not* fsync() on all Append() calls
I20260927 02:39:10.917205 7772 tablet_bootstrap.cc:492] T 75ab554f1fda4a61ace553381f5db0c4 P 95c9bea2bd4044729172198bd0ee9bd4: No bootstrap required, opened a new log
I20260927 02:39:10.917436 7772 ts_tablet_manager.cc:1397] T 75ab554f1fda4a61ace553381f5db0c4 P 95c9bea2bd4044729172198bd0ee9bd4: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 02:39:10.918371 7771 raft_consensus.cc:348] T 75ab554f1fda4a61ace553381f5db0c4 P 6c73821393684ab0961970fdda44e818 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c73821393684ab0961970fdda44e818" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35281 } } peers { permanent_uuid: "95c9bea2bd4044729172198bd0ee9bd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44515 } } peers { permanent_uuid: "bfb91636c48348d1abc15549cd284922" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38069 } }
I20260927 02:39:10.918601 7771 raft_consensus.cc:374] T 75ab554f1fda4a61ace553381f5db0c4 P 6c73821393684ab0961970fdda44e818 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:39:10.918648 7771 raft_consensus.cc:729] T 75ab554f1fda4a61ace553381f5db0c4 P 6c73821393684ab0961970fdda44e818 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c73821393684ab0961970fdda44e818, State: Initialized, Role: FOLLOWER
I20260927 02:39:10.919077 7771 consensus_queue.cc:260] T 75ab554f1fda4a61ace553381f5db0c4 P 6c73821393684ab0961970fdda44e818 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c73821393684ab0961970fdda44e818" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35281 } } peers { permanent_uuid: "95c9bea2bd4044729172198bd0ee9bd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44515 } } peers { permanent_uuid: "bfb91636c48348d1abc15549cd284922" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38069 } }
I20260927 02:39:10.918936 7770 raft_consensus.cc:348] T 75ab554f1fda4a61ace553381f5db0c4 P bfb91636c48348d1abc15549cd284922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c73821393684ab0961970fdda44e818" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35281 } } peers { permanent_uuid: "95c9bea2bd4044729172198bd0ee9bd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44515 } } peers { permanent_uuid: "bfb91636c48348d1abc15549cd284922" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38069 } }
I20260927 02:39:10.919198 7770 raft_consensus.cc:374] T 75ab554f1fda4a61ace553381f5db0c4 P bfb91636c48348d1abc15549cd284922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:39:10.919256 7770 raft_consensus.cc:729] T 75ab554f1fda4a61ace553381f5db0c4 P bfb91636c48348d1abc15549cd284922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfb91636c48348d1abc15549cd284922, State: Initialized, Role: FOLLOWER
I20260927 02:39:10.919754 7464 heartbeater.cc:503] Master 127.0.98.62:35857 was elected leader, sending a full tablet report...
I20260927 02:39:10.919744 7770 consensus_queue.cc:260] T 75ab554f1fda4a61ace553381f5db0c4 P bfb91636c48348d1abc15549cd284922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c73821393684ab0961970fdda44e818" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35281 } } peers { permanent_uuid: "95c9bea2bd4044729172198bd0ee9bd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44515 } } peers { permanent_uuid: "bfb91636c48348d1abc15549cd284922" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38069 } }
I20260927 02:39:10.920697 7770 ts_tablet_manager.cc:1428] T 75ab554f1fda4a61ace553381f5db0c4 P bfb91636c48348d1abc15549cd284922: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260927 02:39:10.921025 7601 heartbeater.cc:503] Master 127.0.98.62:35857 was elected leader, sending a full tablet report...
I20260927 02:39:10.922343 7772 raft_consensus.cc:348] T 75ab554f1fda4a61ace553381f5db0c4 P 95c9bea2bd4044729172198bd0ee9bd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c73821393684ab0961970fdda44e818" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35281 } } peers { permanent_uuid: "95c9bea2bd4044729172198bd0ee9bd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44515 } } peers { permanent_uuid: "bfb91636c48348d1abc15549cd284922" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38069 } }
I20260927 02:39:10.922597 7772 raft_consensus.cc:374] T 75ab554f1fda4a61ace553381f5db0c4 P 95c9bea2bd4044729172198bd0ee9bd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:39:10.922652 7772 raft_consensus.cc:729] T 75ab554f1fda4a61ace553381f5db0c4 P 95c9bea2bd4044729172198bd0ee9bd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95c9bea2bd4044729172198bd0ee9bd4, State: Initialized, Role: FOLLOWER
I20260927 02:39:10.922952 7771 ts_tablet_manager.cc:1428] T 75ab554f1fda4a61ace553381f5db0c4 P 6c73821393684ab0961970fdda44e818: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260927 02:39:10.924782 7772 consensus_queue.cc:260] T 75ab554f1fda4a61ace553381f5db0c4 P 95c9bea2bd4044729172198bd0ee9bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c73821393684ab0961970fdda44e818" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35281 } } peers { permanent_uuid: "95c9bea2bd4044729172198bd0ee9bd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44515 } } peers { permanent_uuid: "bfb91636c48348d1abc15549cd284922" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38069 } }
I20260927 02:39:10.925779 7772 ts_tablet_manager.cc:1428] T 75ab554f1fda4a61ace553381f5db0c4 P 95c9bea2bd4044729172198bd0ee9bd4: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260927 02:39:10.926119 7737 heartbeater.cc:503] Master 127.0.98.62:35857 was elected leader, sending a full tablet report...
W20260927 02:39:10.968294 7467 tablet.cc:2369] T 75ab554f1fda4a61ace553381f5db0c4 P 6c73821393684ab0961970fdda44e818: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:39:11.096436 7740 tablet.cc:2369] T 75ab554f1fda4a61ace553381f5db0c4 P 95c9bea2bd4044729172198bd0ee9bd4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:39:11.132288 7777 raft_consensus.cc:482] T 75ab554f1fda4a61ace553381f5db0c4 P bfb91636c48348d1abc15549cd284922 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:39:11.132485 7777 raft_consensus.cc:504] T 75ab554f1fda4a61ace553381f5db0c4 P bfb91636c48348d1abc15549cd284922 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c73821393684ab0961970fdda44e818" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35281 } } peers { permanent_uuid: "95c9bea2bd4044729172198bd0ee9bd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44515 } } peers { permanent_uuid: "bfb91636c48348d1abc15549cd284922" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38069 } }
I20260927 02:39:11.133701 7777 leader_election.cc:288] T 75ab554f1fda4a61ace553381f5db0c4 P bfb91636c48348d1abc15549cd284922 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c73821393684ab0961970fdda44e818 (127.0.98.1:35281), 95c9bea2bd4044729172198bd0ee9bd4 (127.0.98.3:44515)
I20260927 02:39:11.137337 7605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:11.133880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35281 (local address 127.0.98.2:38299)
0927 02:39:11.134010 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:11.134025 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:11.134118 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:11.134470 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:11.134473 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:11.134491 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:39:11.134621 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:11.134627 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:11.135367 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:11.135370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:11.136247 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:11.136256 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:11.136905 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:11.136909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:11.137010 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:11.137035 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 02:39:11.137105 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 02:39:11.137213 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":23}
I20260927 02:39:11.137356 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:11.134095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38299 (local address 127.0.98.1:35281)
0927 02:39:11.134261 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:11.134266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:11.134280 (+ 14us) server_negotiation.cc:407] Connection header received
0927 02:39:11.134329 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:11.134333 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:11.134381 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:11.134440 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:39:11.134725 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:11.135254 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:11.136375 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:11.136799 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:11.136882 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:11.136934 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:11.137144 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 02:39:11.137233 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 02:39:11.137554 7607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:11.133982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44515 (local address 127.0.98.2:44119)
0927 02:39:11.134183 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:11.134197 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:11.134307 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:11.134545 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:11.134549 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:11.134566 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:39:11.134746 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:11.134754 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:11.135531 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:11.135534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:11.136383 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:11.136393 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:11.137141 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:11.137145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:11.137255 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:11.137276 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 02:39:11.137350 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 02:39:11.137415 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":35}
I20260927 02:39:11.137554 7769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:11.134057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44119 (local address 127.0.98.3:44515)
0927 02:39:11.134171 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:11.134175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:11.134280 (+ 105us) server_negotiation.cc:407] Connection header received
0927 02:39:11.134392 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:11.134394 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:11.134442 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:11.134515 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:39:11.134880 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:11.135399 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:11.136529 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:11.137004 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:11.137113 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:11.137170 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:11.137377 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 02:39:11.137449 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 02:39:11.138099 7418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ab554f1fda4a61ace553381f5db0c4" candidate_uuid: "bfb91636c48348d1abc15549cd284922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c73821393684ab0961970fdda44e818" is_pre_election: true
I20260927 02:39:11.138127 7691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ab554f1fda4a61ace553381f5db0c4" candidate_uuid: "bfb91636c48348d1abc15549cd284922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95c9bea2bd4044729172198bd0ee9bd4" is_pre_election: true
I20260927 02:39:11.138409 7418 raft_consensus.cc:2454] T 75ab554f1fda4a61ace553381f5db0c4 P 6c73821393684ab0961970fdda44e818 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfb91636c48348d1abc15549cd284922 in term 0.
I20260927 02:39:11.138409 7691 raft_consensus.cc:2454] T 75ab554f1fda4a61ace553381f5db0c4 P 95c9bea2bd4044729172198bd0ee9bd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfb91636c48348d1abc15549cd284922 in term 0.
I20260927 02:39:11.138867 7488 leader_election.cc:302] T 75ab554f1fda4a61ace553381f5db0c4 P bfb91636c48348d1abc15549cd284922 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6c73821393684ab0961970fdda44e818, 95c9bea2bd4044729172198bd0ee9bd4, bfb91636c48348d1abc15549cd284922; no voters:
I20260927 02:39:11.139132 7777 raft_consensus.cc:2781] T 75ab554f1fda4a61ace553381f5db0c4 P bfb91636c48348d1abc15549cd284922 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:39:11.139219 7777 raft_consensus.cc:482] T 75ab554f1fda4a61ace553381f5db0c4 P bfb91636c48348d1abc15549cd284922 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:39:11.139261 7777 raft_consensus.cc:3037] T 75ab554f1fda4a61ace553381f5db0c4 P bfb91636c48348d1abc15549cd284922 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:39:11.140254 7777 raft_consensus.cc:504] T 75ab554f1fda4a61ace553381f5db0c4 P bfb91636c48348d1abc15549cd284922 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c73821393684ab0961970fdda44e818" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35281 } } peers { permanent_uuid: "95c9bea2bd4044729172198bd0ee9bd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44515 } } peers { permanent_uuid: "bfb91636c48348d1abc15549cd284922" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38069 } }
I20260927 02:39:11.140560 7777 leader_election.cc:288] T 75ab554f1fda4a61ace553381f5db0c4 P bfb91636c48348d1abc15549cd284922 [CANDIDATE]: Term 1 election: Requested vote from peers 6c73821393684ab0961970fdda44e818 (127.0.98.1:35281), 95c9bea2bd4044729172198bd0ee9bd4 (127.0.98.3:44515)
I20260927 02:39:11.140936 7691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ab554f1fda4a61ace553381f5db0c4" candidate_uuid: "bfb91636c48348d1abc15549cd284922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95c9bea2bd4044729172198bd0ee9bd4"
I20260927 02:39:11.140942 7418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ab554f1fda4a61ace553381f5db0c4" candidate_uuid: "bfb91636c48348d1abc15549cd284922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c73821393684ab0961970fdda44e818"
I20260927 02:39:11.141041 7418 raft_consensus.cc:3037] T 75ab554f1fda4a61ace553381f5db0c4 P 6c73821393684ab0961970fdda44e818 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:39:11.141042 7691 raft_consensus.cc:3037] T 75ab554f1fda4a61ace553381f5db0c4 P 95c9bea2bd4044729172198bd0ee9bd4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:39:11.142000 7418 raft_consensus.cc:2454] T 75ab554f1fda4a61ace553381f5db0c4 P 6c73821393684ab0961970fdda44e818 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfb91636c48348d1abc15549cd284922 in term 1.
I20260927 02:39:11.142059 7691 raft_consensus.cc:2454] T 75ab554f1fda4a61ace553381f5db0c4 P 95c9bea2bd4044729172198bd0ee9bd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfb91636c48348d1abc15549cd284922 in term 1.
I20260927 02:39:11.142333 7490 leader_election.cc:302] T 75ab554f1fda4a61ace553381f5db0c4 P bfb91636c48348d1abc15549cd284922 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c73821393684ab0961970fdda44e818, bfb91636c48348d1abc15549cd284922; no voters:
I20260927 02:39:11.142499 7777 raft_consensus.cc:2781] T 75ab554f1fda4a61ace553381f5db0c4 P bfb91636c48348d1abc15549cd284922 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:39:11.142668 7777 raft_consensus.cc:686] T 75ab554f1fda4a61ace553381f5db0c4 P bfb91636c48348d1abc15549cd284922 [term 1 LEADER]: Becoming Leader. State: Replica: bfb91636c48348d1abc15549cd284922, State: Running, Role: LEADER
I20260927 02:39:11.143153 7777 consensus_queue.cc:237] T 75ab554f1fda4a61ace553381f5db0c4 P bfb91636c48348d1abc15549cd284922 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c73821393684ab0961970fdda44e818" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35281 } } peers { permanent_uuid: "95c9bea2bd4044729172198bd0ee9bd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44515 } } peers { permanent_uuid: "bfb91636c48348d1abc15549cd284922" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38069 } }
I20260927 02:39:11.146520 6501 catalog_manager.cc:5495] T 75ab554f1fda4a61ace553381f5db0c4 P bfb91636c48348d1abc15549cd284922 reported cstate change: term changed from 0 to 1, leader changed from <none> to bfb91636c48348d1abc15549cd284922 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "bfb91636c48348d1abc15549cd284922" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c73821393684ab0961970fdda44e818" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95c9bea2bd4044729172198bd0ee9bd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfb91636c48348d1abc15549cd284922" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38069 } health_report { overall_health: HEALTHY } } }
W20260927 02:39:11.152792 7604 tablet.cc:2369] T 75ab554f1fda4a61ace553381f5db0c4 P bfb91636c48348d1abc15549cd284922: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:39:11.192659 7768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:11.189263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48380 (local address 127.0.98.2:38069)
0927 02:39:11.189381 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:11.189385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:11.189399 (+ 14us) server_negotiation.cc:407] Connection header received
0927 02:39:11.189503 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:11.189505 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:11.189559 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:11.189653 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:39:11.189884 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:11.190461 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:11.191353 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:11.191568 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:11.191650 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:11.191916 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:39:11.192342 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:39:11.192433 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:11.192497 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 02:39:11.192546 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 02:39:11.198411 7691 raft_consensus.cc:1261] T 75ab554f1fda4a61ace553381f5db0c4 P 95c9bea2bd4044729172198bd0ee9bd4 [term 1 FOLLOWER]: Refusing update from remote peer bfb91636c48348d1abc15549cd284922: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:39:11.198421 7418 raft_consensus.cc:1261] T 75ab554f1fda4a61ace553381f5db0c4 P 6c73821393684ab0961970fdda44e818 [term 1 FOLLOWER]: Refusing update from remote peer bfb91636c48348d1abc15549cd284922: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:39:11.199086 7779 consensus_queue.cc:1035] T 75ab554f1fda4a61ace553381f5db0c4 P bfb91636c48348d1abc15549cd284922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c73821393684ab0961970fdda44e818" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:39:11.199250 7777 consensus_queue.cc:1035] T 75ab554f1fda4a61ace553381f5db0c4 P bfb91636c48348d1abc15549cd284922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95c9bea2bd4044729172198bd0ee9bd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:39:11.205235 7782 mvcc.cc:205] Tried to move back new op lower bound from 7333792772901900288 to 7333792772685139968. Current Snapshot: MvccSnapshot[applied={T|T < 7333792772901900288}]
I20260927 02:39:11.210335 7784 mvcc.cc:205] Tried to move back new op lower bound from 7333792772901900288 to 7333792772685139968. Current Snapshot: MvccSnapshot[applied={T|T < 7333792772901900288}]
I20260927 02:39:11.210495 7783 mvcc.cc:205] Tried to move back new op lower bound from 7333792772901900288 to 7333792772685139968. Current Snapshot: MvccSnapshot[applied={T|T < 7333792772901900288 or (T in {7333792772901900288})}]
I20260927 02:39:11.217113 6502 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:39:11.217274 6502 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:39:11.221168 6502 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 372b9126cbeb4dbdbd3b50c5bfa23a8f: Sending DeleteTablet for 3 replicas of tablet 75ab554f1fda4a61ace553381f5db0c4
I20260927 02:39:11.222003 7398 tablet_service.cc:1514] Processing DeleteTablet for tablet 75ab554f1fda4a61ace553381f5db0c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:39:11 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39420
I20260927 02:39:11.222359 7671 tablet_service.cc:1514] Processing DeleteTablet for tablet 75ab554f1fda4a61ace553381f5db0c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:39:11 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49070
I20260927 02:39:11.222504 7517 tablet_service.cc:1514] Processing DeleteTablet for tablet 75ab554f1fda4a61ace553381f5db0c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:39:11 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48376
I20260927 02:39:11.222935 7794 tablet_replica.cc:330] T 75ab554f1fda4a61ace553381f5db0c4 P bfb91636c48348d1abc15549cd284922: stopping tablet replica
I20260927 02:39:11.223220 7793 tablet_replica.cc:330] T 75ab554f1fda4a61ace553381f5db0c4 P 95c9bea2bd4044729172198bd0ee9bd4: stopping tablet replica
I20260927 02:39:11.223353 7794 raft_consensus.cc:2229] T 75ab554f1fda4a61ace553381f5db0c4 P bfb91636c48348d1abc15549cd284922 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:39:11.223523 7793 raft_consensus.cc:2229] T 75ab554f1fda4a61ace553381f5db0c4 P 95c9bea2bd4044729172198bd0ee9bd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:39:11.223862 7793 raft_consensus.cc:2258] T 75ab554f1fda4a61ace553381f5db0c4 P 95c9bea2bd4044729172198bd0ee9bd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:39:11.223863 7794 raft_consensus.cc:2258] T 75ab554f1fda4a61ace553381f5db0c4 P bfb91636c48348d1abc15549cd284922 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:39:11.225497 7794 ts_tablet_manager.cc:1905] T 75ab554f1fda4a61ace553381f5db0c4 P bfb91636c48348d1abc15549cd284922: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:39:11.225497 7793 ts_tablet_manager.cc:1905] T 75ab554f1fda4a61ace553381f5db0c4 P 95c9bea2bd4044729172198bd0ee9bd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:39:11.225675 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 6470
I20260927 02:39:11.228668 7793 ts_tablet_manager.cc:1918] T 75ab554f1fda4a61ace553381f5db0c4 P 95c9bea2bd4044729172198bd0ee9bd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:39:11.228828 7793 log.cc:1196] T 75ab554f1fda4a61ace553381f5db0c4 P 95c9bea2bd4044729172198bd0ee9bd4: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-2/wal/wals/75ab554f1fda4a61ace553381f5db0c4
I20260927 02:39:11.229120 7794 ts_tablet_manager.cc:1918] T 75ab554f1fda4a61ace553381f5db0c4 P bfb91636c48348d1abc15549cd284922: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:39:11.229215 7794 log.cc:1196] T 75ab554f1fda4a61ace553381f5db0c4 P bfb91636c48348d1abc15549cd284922: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-1/wal/wals/75ab554f1fda4a61ace553381f5db0c4
I20260927 02:39:11.229317 7793 ts_tablet_manager.cc:1939] T 75ab554f1fda4a61ace553381f5db0c4 P 95c9bea2bd4044729172198bd0ee9bd4: Deleting consensus metadata
I20260927 02:39:11.229504 7794 ts_tablet_manager.cc:1939] T 75ab554f1fda4a61ace553381f5db0c4 P bfb91636c48348d1abc15549cd284922: Deleting consensus metadata
W20260927 02:39:11.232362 7488 connection.cc:535] server connection from 127.0.0.1:48376 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 02:39:11.232447 7624 connection.cc:535] server connection from 127.0.0.1:49070 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 02:39:11.233573 7792 tablet_replica.cc:330] T 75ab554f1fda4a61ace553381f5db0c4 P 6c73821393684ab0961970fdda44e818: stopping tablet replica
I20260927 02:39:11.233918 7792 raft_consensus.cc:2229] T 75ab554f1fda4a61ace553381f5db0c4 P 6c73821393684ab0961970fdda44e818 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:39:11.234292 7792 raft_consensus.cc:2258] T 75ab554f1fda4a61ace553381f5db0c4 P 6c73821393684ab0961970fdda44e818 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:39:11.236334 7792 ts_tablet_manager.cc:1905] T 75ab554f1fda4a61ace553381f5db0c4 P 6c73821393684ab0961970fdda44e818: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:39:11.239158 7792 ts_tablet_manager.cc:1918] T 75ab554f1fda4a61ace553381f5db0c4 P 6c73821393684ab0961970fdda44e818: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:39:11.239315 7792 log.cc:1196] T 75ab554f1fda4a61ace553381f5db0c4 P 6c73821393684ab0961970fdda44e818: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-0/wal/wals/75ab554f1fda4a61ace553381f5db0c4
I20260927 02:39:11.239703 7792 ts_tablet_manager.cc:1939] T 75ab554f1fda4a61ace553381f5db0c4 P 6c73821393684ab0961970fdda44e818: Deleting consensus metadata
W20260927 02:39:11.240950 7352 connection.cc:441] server connection from 127.0.0.1:39420 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39420 (request call id 1) could send its response
I20260927 02:39:11.244387 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:11.225203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52956 (local address 127.0.98.61:40345)
0927 02:39:11.225316 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:11.225320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:11.234720 (+ 9400us) server_negotiation.cc:407] Connection header received
0927 02:39:11.234843 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:11.234846 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:11.234904 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:11.234989 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:11.235822 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:11.236568 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:11.237869 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:11.238195 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:11.238346 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:11.240129 (+ 1783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:11.240133 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:11.240136 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:11.240163 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:11.242523 (+ 2360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:11.243017 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:11.243021 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:11.243022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:11.243110 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:11.243472 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:11.243475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:11.243477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:11.243686 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:11.243803 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:11.244098 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 02:39:11.244220 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 02:39:11.248013 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:11.235916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47800 (local address 127.0.98.60:33141)
0927 02:39:11.236740 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:11.236744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:11.236757 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:39:11.236796 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:11.236799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:11.236842 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:11.236908 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:11.239161 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:11.239967 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:11.241271 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:11.241588 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:11.241676 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:11.243147 (+ 1471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:11.243151 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:11.243154 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:11.243179 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:11.245927 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:11.246445 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:11.246448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:11.246450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:11.246534 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:11.246893 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:11.246896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:11.246898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:11.247089 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:11.247211 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:11.247748 (+ 537us) server_negotiation.cc:299] Negotiation successful
0927 02:39:11.247857 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":725}
I20260927 02:39:11.251364 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:11.237043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52968 (local address 127.0.98.61:40345)
0927 02:39:11.237161 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:11.237165 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:11.241500 (+ 4335us) server_negotiation.cc:407] Connection header received
0927 02:39:11.241575 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:11.241578 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:11.241625 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:11.241708 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:11.242824 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:11.243859 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:11.245770 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:11.246107 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:11.246175 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:11.247856 (+ 1681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:11.247860 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:11.247862 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:11.247885 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:11.249631 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:11.250100 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:11.250103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:11.250105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:11.250178 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:11.250500 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:11.250504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:11.250506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:11.250734 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:11.250865 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:11.251089 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 02:39:11.251205 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 02:39:11.252931 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:11.224266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47790 (local address 127.0.98.60:33141)
0927 02:39:11.224652 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:11.224657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:11.244496 (+ 19839us) server_negotiation.cc:407] Connection header received
0927 02:39:11.244650 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:11.244653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:11.244701 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:11.244749 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:11.245454 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:11.246167 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:11.247265 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:11.247577 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:11.247649 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:11.249907 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:11.249911 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:11.249913 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:11.249936 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:11.251383 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:11.251859 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:11.251862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:11.251864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:11.251937 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:11.252164 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:11.252168 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:11.252170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:11.252353 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:11.252456 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:11.252659 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 02:39:11.252775 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312}
I20260927 02:39:12.209503 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:12.209050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35857 (local address 127.0.98.1:42047)
0927 02:39:12.209296 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:12.209390 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":80,"threads_started":1}
W20260927 02:39:12.209892 7464 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35857 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:35857: connect: Connection refused (error 111)
I20260927 02:39:12.213064 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:12.212597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35857 (local address 127.0.98.2:37859)
0927 02:39:12.212888 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:12.212980 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":75,"threads_started":1}
W20260927 02:39:12.213599 7601 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35857 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:35857: connect: Connection refused (error 111)
I20260927 02:39:12.213847 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:12.213470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35857 (local address 127.0.98.3:37293)
0927 02:39:12.213672 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:12.213752 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":66,"threads_started":1}
W20260927 02:39:12.214165 7737 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35857 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:35857: connect: Connection refused (error 111)
I20260927 02:39:12.722330 7805 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372b9126cbeb4dbdbd3b50c5bfa23a8f)
I20260927 02:39:12.722515 7805 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } } peers { permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40345 } attrs { promote: false } } peers { permanent_uuid: "50ce6beeca7d4e7186a77b7c34dcb011" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33141 } attrs { promote: false } }
I20260927 02:39:12.723452 7805 leader_election.cc:288] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372b9126cbeb4dbdbd3b50c5bfa23a8f (127.0.98.62:35857), 6f4dad5a3ce0416eb97e8a4be06dff38 (127.0.98.61:40345)
I20260927 02:39:12.723517 7806 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372b9126cbeb4dbdbd3b50c5bfa23a8f)
I20260927 02:39:12.723897 7806 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } } peers { permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40345 } attrs { promote: false } } peers { permanent_uuid: "50ce6beeca7d4e7186a77b7c34dcb011" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33141 } attrs { promote: false } }
I20260927 02:39:12.724084 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:12.723588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35857 (local address 127.0.0.1:48934)
0927 02:39:12.723896 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:12.723980 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":103,"threads_started":1}
W20260927 02:39:12.724249 7270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35857: connect: Connection refused (error 111)
I20260927 02:39:12.725023 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:12.724785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35857 (local address 127.0.0.1:48936)
0927 02:39:12.724910 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:12.724957 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 02:39:12.725065 7806 leader_election.cc:288] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372b9126cbeb4dbdbd3b50c5bfa23a8f (127.0.98.62:35857), 50ce6beeca7d4e7186a77b7c34dcb011 (127.0.98.60:33141)
W20260927 02:39:12.725238 7270 leader_election.cc:334] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 372b9126cbeb4dbdbd3b50c5bfa23a8f (127.0.98.62:35857): Network error: Client connection negotiation failed: client connection to 127.0.98.62:35857: connect: Connection refused (error 111)
I20260927 02:39:12.725488 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:12.725143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35857 (local address 127.0.0.1:48942)
0927 02:39:12.725341 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:12.725406 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":68,"threads_started":1}
W20260927 02:39:12.725646 7088 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35857: connect: Connection refused (error 111)
I20260927 02:39:12.726392 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:12.726167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35857 (local address 127.0.0.1:48954)
0927 02:39:12.726279 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:12.726329 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 02:39:12.726521 7088 leader_election.cc:334] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 372b9126cbeb4dbdbd3b50c5bfa23a8f (127.0.98.62:35857): Network error: Client connection negotiation failed: client connection to 127.0.98.62:35857: connect: Connection refused (error 111)
I20260927 02:39:12.728257 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:12.723675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40345 (local address 127.0.0.1:52982)
0927 02:39:12.723971 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:12.723986 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:12.724091 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:12.724841 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:12.724844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:12.724863 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:39:12.724956 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:12.724961 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:12.726282 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:12.726285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:12.727261 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:12.727271 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:12.727898 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:12.727902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:12.727988 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:12.728014 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 02:39:12.728089 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 02:39:12.728159 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":64,"threads_started":1}
I20260927 02:39:12.728698 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:12.723882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52982 (local address 127.0.98.61:40345)
0927 02:39:12.724557 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:12.724562 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:12.724579 (+ 17us) server_negotiation.cc:407] Connection header received
0927 02:39:12.724630 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:12.724633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:12.724682 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:12.724787 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:39:12.725134 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:12.725979 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:12.727386 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:12.727788 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:12.727879 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:12.727925 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:12.728503 (+ 578us) server_negotiation.cc:299] Negotiation successful
0927 02:39:12.728577 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":87,"threads_started":1}
I20260927 02:39:12.729064 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:12.725426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33141 (local address 127.0.0.1:47802)
0927 02:39:12.725606 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:12.725618 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:12.725697 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:12.726100 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:12.726104 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:12.726119 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:39:12.726195 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:12.726200 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:12.727093 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:12.727097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:12.727998 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:12.728007 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:12.728705 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:12.728708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:12.728792 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:12.728818 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 02:39:12.728889 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 02:39:12.728954 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":54,"threads_started":1}
I20260927 02:39:12.729072 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:12.725581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47802 (local address 127.0.98.60:33141)
0927 02:39:12.725878 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:12.725886 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:12.725899 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:39:12.725939 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:12.725942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:12.725983 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:12.726314 (+ 331us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:39:12.726400 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:12.726995 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:12.728117 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:12.728606 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:12.728699 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:12.728752 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:12.728905 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 02:39:12.728971 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":87,"threads_started":1}
I20260927 02:39:12.729092 7113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "50ce6beeca7d4e7186a77b7c34dcb011" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" is_pre_election: true
I20260927 02:39:12.729280 7113 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50ce6beeca7d4e7186a77b7c34dcb011 in term 1.
I20260927 02:39:12.729471 7294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "50ce6beeca7d4e7186a77b7c34dcb011" is_pre_election: true
I20260927 02:39:12.729619 7272 leader_election.cc:302] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 50ce6beeca7d4e7186a77b7c34dcb011, 6f4dad5a3ce0416eb97e8a4be06dff38; no voters: 372b9126cbeb4dbdbd3b50c5bfa23a8f
I20260927 02:39:12.729637 7294 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f4dad5a3ce0416eb97e8a4be06dff38 in term 1.
I20260927 02:39:12.729862 7805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 02:39:12.729929 7805 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 372b9126cbeb4dbdbd3b50c5bfa23a8f)
I20260927 02:39:12.729971 7805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011 [term 1 FOLLOWER]: Advancing to term 2
I20260927 02:39:12.730067 7088 leader_election.cc:302] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 50ce6beeca7d4e7186a77b7c34dcb011, 6f4dad5a3ce0416eb97e8a4be06dff38; no voters: 372b9126cbeb4dbdbd3b50c5bfa23a8f
I20260927 02:39:12.730275 7806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 02:39:12.730337 7806 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 372b9126cbeb4dbdbd3b50c5bfa23a8f)
I20260927 02:39:12.730392 7806 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [term 1 FOLLOWER]: Advancing to term 2
I20260927 02:39:12.731088 7805 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } } peers { permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40345 } attrs { promote: false } } peers { permanent_uuid: "50ce6beeca7d4e7186a77b7c34dcb011" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33141 } attrs { promote: false } }
I20260927 02:39:12.731344 7806 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } } peers { permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40345 } attrs { promote: false } } peers { permanent_uuid: "50ce6beeca7d4e7186a77b7c34dcb011" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33141 } attrs { promote: false } }
I20260927 02:39:12.731449 7805 leader_election.cc:288] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011 [CANDIDATE]: Term 2 election: Requested vote from peers 372b9126cbeb4dbdbd3b50c5bfa23a8f (127.0.98.62:35857), 6f4dad5a3ce0416eb97e8a4be06dff38 (127.0.98.61:40345)
I20260927 02:39:12.731684 7806 leader_election.cc:288] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [CANDIDATE]: Term 2 election: Requested vote from peers 372b9126cbeb4dbdbd3b50c5bfa23a8f (127.0.98.62:35857), 50ce6beeca7d4e7186a77b7c34dcb011 (127.0.98.60:33141)
I20260927 02:39:12.731871 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:12.731572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35857 (local address 127.0.0.1:48962)
0927 02:39:12.731686 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:12.731743 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 02:39:12.731834 7113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "50ce6beeca7d4e7186a77b7c34dcb011" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38"
I20260927 02:39:12.731997 7113 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 50ce6beeca7d4e7186a77b7c34dcb011 in current term 2: Already voted for candidate 6f4dad5a3ce0416eb97e8a4be06dff38 in this term.
I20260927 02:39:12.732095 7294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "50ce6beeca7d4e7186a77b7c34dcb011"
I20260927 02:39:12.732246 7294 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 6f4dad5a3ce0416eb97e8a4be06dff38 in current term 2: Already voted for candidate 50ce6beeca7d4e7186a77b7c34dcb011 in this term.
I20260927 02:39:12.732456 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:12.731772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35857 (local address 127.0.0.1:48974)
0927 02:39:12.731985 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:12.732382 (+ 397us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
I20260927 02:39:12.732626 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:12.732365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35857 (local address 127.0.0.1:48988)
0927 02:39:12.732459 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:12.732574 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 02:39:12.732760 7270 leader_election.cc:334] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 372b9126cbeb4dbdbd3b50c5bfa23a8f (127.0.98.62:35857): Network error: Client connection negotiation failed: client connection to 127.0.98.62:35857: connect: Connection refused (error 111)
I20260927 02:39:12.732824 7270 leader_election.cc:302] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 50ce6beeca7d4e7186a77b7c34dcb011; no voters: 372b9126cbeb4dbdbd3b50c5bfa23a8f, 6f4dad5a3ce0416eb97e8a4be06dff38
I20260927 02:39:12.732949 7805 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 02:39:12.733157 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:12.732937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35857 (local address 127.0.0.1:48998)
0927 02:39:12.733063 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:12.733110 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 02:39:12.733249 7088 leader_election.cc:334] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 372b9126cbeb4dbdbd3b50c5bfa23a8f (127.0.98.62:35857): Network error: Client connection negotiation failed: client connection to 127.0.98.62:35857: connect: Connection refused (error 111)
I20260927 02:39:12.733309 7088 leader_election.cc:302] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f4dad5a3ce0416eb97e8a4be06dff38; no voters: 372b9126cbeb4dbdbd3b50c5bfa23a8f, 50ce6beeca7d4e7186a77b7c34dcb011
I20260927 02:39:12.733431 7806 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 02:39:13.210965 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:13.210513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35857 (local address 127.0.98.1:49649)
0927 02:39:13.210786 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:13.210856 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":83,"threads_started":1}
I20260927 02:39:13.214591 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:13.214123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35857 (local address 127.0.98.2:40591)
0927 02:39:13.214400 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:13.214466 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":78,"threads_started":1}
I20260927 02:39:13.215093 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:13.214641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35857 (local address 127.0.98.3:57035)
0927 02:39:13.214940 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:13.215008 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":64,"threads_started":1}
I20260927 02:39:14.212327 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:14.211915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35857 (local address 127.0.98.1:40689)
0927 02:39:14.212158 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:14.212227 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":86,"threads_started":1}
I20260927 02:39:14.215867 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:14.215427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35857 (local address 127.0.98.2:51205)
0927 02:39:14.215705 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:14.215769 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":82,"threads_started":1}
I20260927 02:39:14.216254 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:14.215625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35857 (local address 127.0.98.3:53431)
0927 02:39:14.215895 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:14.216165 (+ 270us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":57,"threads_started":1}
I20260927 02:39:14.470523 7821 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:39:14.470655 7821 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } } peers { permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40345 } attrs { promote: false } } peers { permanent_uuid: "50ce6beeca7d4e7186a77b7c34dcb011" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33141 } attrs { promote: false } }
I20260927 02:39:14.471012 7821 leader_election.cc:288] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 372b9126cbeb4dbdbd3b50c5bfa23a8f (127.0.98.62:35857), 50ce6beeca7d4e7186a77b7c34dcb011 (127.0.98.60:33141)
I20260927 02:39:14.471639 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:14.471232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35857 (local address 127.0.0.1:49224)
0927 02:39:14.471482 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:14.471553 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":75,"threads_started":1}
I20260927 02:39:14.471756 7294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "50ce6beeca7d4e7186a77b7c34dcb011" is_pre_election: true
I20260927 02:39:14.471922 7294 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f4dad5a3ce0416eb97e8a4be06dff38 in term 2.
I20260927 02:39:14.472401 7088 leader_election.cc:302] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50ce6beeca7d4e7186a77b7c34dcb011, 6f4dad5a3ce0416eb97e8a4be06dff38; no voters:
I20260927 02:39:14.472478 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:14.472253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35857 (local address 127.0.0.1:49228)
0927 02:39:14.472353 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:14.472400 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 02:39:14.472553 7088 leader_election.cc:334] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 372b9126cbeb4dbdbd3b50c5bfa23a8f (127.0.98.62:35857): Network error: Client connection negotiation failed: client connection to 127.0.98.62:35857: connect: Connection refused (error 111)
I20260927 02:39:14.472584 7821 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 02:39:14.472657 7821 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:39:14.472708 7821 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [term 2 FOLLOWER]: Advancing to term 3
I20260927 02:39:14.473662 7821 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } } peers { permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40345 } attrs { promote: false } } peers { permanent_uuid: "50ce6beeca7d4e7186a77b7c34dcb011" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33141 } attrs { promote: false } }
I20260927 02:39:14.473941 7821 leader_election.cc:288] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [CANDIDATE]: Term 3 election: Requested vote from peers 372b9126cbeb4dbdbd3b50c5bfa23a8f (127.0.98.62:35857), 50ce6beeca7d4e7186a77b7c34dcb011 (127.0.98.60:33141)
I20260927 02:39:14.474313 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:14.474064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35857 (local address 127.0.0.1:49242)
0927 02:39:14.474161 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:14.474205 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 02:39:14.474447 7294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "50ce6beeca7d4e7186a77b7c34dcb011"
I20260927 02:39:14.474551 7294 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011 [term 2 FOLLOWER]: Advancing to term 3
I20260927 02:39:14.475387 7294 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f4dad5a3ce0416eb97e8a4be06dff38 in term 3.
I20260927 02:39:14.475687 7088 leader_election.cc:302] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50ce6beeca7d4e7186a77b7c34dcb011, 6f4dad5a3ce0416eb97e8a4be06dff38; no voters:
I20260927 02:39:14.475829 7821 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 02:39:14.475993 7821 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [term 3 LEADER]: Becoming Leader. State: Replica: 6f4dad5a3ce0416eb97e8a4be06dff38, State: Running, Role: LEADER
I20260927 02:39:14.476123 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:14.475913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35857 (local address 127.0.0.1:49244)
0927 02:39:14.476011 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:14.476058 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 02:39:14.476254 7088 leader_election.cc:334] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 372b9126cbeb4dbdbd3b50c5bfa23a8f (127.0.98.62:35857): Network error: Client connection negotiation failed: client connection to 127.0.98.62:35857: connect: Connection refused (error 111)
I20260927 02:39:14.476253 7821 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } } peers { permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40345 } attrs { promote: false } } peers { permanent_uuid: "50ce6beeca7d4e7186a77b7c34dcb011" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33141 } attrs { promote: false } }
I20260927 02:39:14.477257 7824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f4dad5a3ce0416eb97e8a4be06dff38. Latest consensus state: current_term: 3 leader_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } } peers { permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40345 } attrs { promote: false } } peers { permanent_uuid: "50ce6beeca7d4e7186a77b7c34dcb011" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33141 } attrs { promote: false } } }
I20260927 02:39:14.477345 7824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [sys.catalog]: This master's current role is: LEADER
I20260927 02:39:14.477625 7826 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:39:14.479770 7826 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:39:14.480127 7826 catalog_manager.cc:1259] Loaded cluster ID: 7432986bc1b74b229d125db051eba32d
I20260927 02:39:14.480170 7826 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:39:14.480657 7826 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:39:14.480989 7826 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38: Loaded TSK: 0
I20260927 02:39:14.481604 7826 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:39:14.538688 7095 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52968:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:39:14.541805 7095 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:39:14.544899 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:14.544672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35857 (local address 127.0.0.1:49252)
0927 02:39:14.544780 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:14.544833 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 02:39:14.545248 7294 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011 [term 3 FOLLOWER]: Refusing update from remote peer 6f4dad5a3ce0416eb97e8a4be06dff38: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260927 02:39:14.545607 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:14.545374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35857 (local address 127.0.0.1:49264)
0927 02:39:14.545474 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:14.545528 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 02:39:14.545709 7821 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50ce6beeca7d4e7186a77b7c34dcb011" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260927 02:39:14.545871 7088 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 -> Peer 372b9126cbeb4dbdbd3b50c5bfa23a8f (127.0.98.62:35857): Couldn't send request to peer 372b9126cbeb4dbdbd3b50c5bfa23a8f. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 02:39:14.547111 7828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f4dad5a3ce0416eb97e8a4be06dff38. Latest consensus state: current_term: 3 leader_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } } peers { permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40345 } attrs { promote: false } } peers { permanent_uuid: "50ce6beeca7d4e7186a77b7c34dcb011" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33141 } attrs { promote: false } } }
I20260927 02:39:14.547222 7828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:39:14.548337 7824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } } peers { permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40345 } attrs { promote: false } } peers { permanent_uuid: "50ce6beeca7d4e7186a77b7c34dcb011" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33141 } attrs { promote: false } } }
I20260927 02:39:14.548511 7824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38 [sys.catalog]: This master's current role is: LEADER
I20260927 02:39:14.548556 7828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "372b9126cbeb4dbdbd3b50c5bfa23a8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35857 } } peers { permanent_uuid: "6f4dad5a3ce0416eb97e8a4be06dff38" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40345 } attrs { promote: false } } peers { permanent_uuid: "50ce6beeca7d4e7186a77b7c34dcb011" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33141 } attrs { promote: false } } }
I20260927 02:39:14.548679 7828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50ce6beeca7d4e7186a77b7c34dcb011 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:39:14.557915 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:14.553776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44515 (local address 127.0.0.1:49084)
0927 02:39:14.553882 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:14.553896 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:14.553983 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:14.554647 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:14.554650 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:14.554667 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:39:14.554776 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:14.554782 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:14.555864 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:14.555866 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:14.556770 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:14.556778 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:14.557537 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:14.557541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:14.557627 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:14.557653 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 02:39:14.557698 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 02:39:14.557768 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":1}
I20260927 02:39:14.558425 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:14.553945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49084 (local address 127.0.98.3:44515)
0927 02:39:14.554399 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:14.554404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:14.554418 (+ 14us) server_negotiation.cc:407] Connection header received
0927 02:39:14.554468 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:14.554471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:14.554516 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:14.554883 (+ 367us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:39:14.554990 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:14.555750 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:14.556877 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:14.557420 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:14.558087 (+ 667us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:14.558162 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:14.558198 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 02:39:14.558290 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":81,"threads_started":1}
I20260927 02:39:14.558954 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:14.554223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35281 (local address 127.0.0.1:39426)
0927 02:39:14.555243 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:14.555255 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:14.555346 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:14.555631 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:14.555634 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:14.555650 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:39:14.555731 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:14.555736 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:14.556461 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:14.556464 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:14.557569 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:14.557578 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:14.558448 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:14.558452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:14.558542 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:14.558564 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 02:39:14.558722 (+ 158us) client_negotiation.cc:240] Negotiation successful
0927 02:39:14.558806 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":966,"mutex_wait_us":302,"spinlock_wait_cycles":1280,"thread_start_us":79,"threads_started":1}
I20260927 02:39:14.558959 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:14.554409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39426 (local address 127.0.98.1:35281)
0927 02:39:14.554886 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:14.554890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:14.555418 (+ 528us) server_negotiation.cc:407] Connection header received
0927 02:39:14.555467 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:14.555470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:14.555511 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:14.555598 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:39:14.555836 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:14.556348 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:14.557704 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:14.558348 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:14.558453 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:14.558510 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:14.558739 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 02:39:14.558822 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":90,"threads_started":1}
I20260927 02:39:14.559176 7671 tablet_service.cc:1467] Processing CreateTablet for tablet 975c91c2cf1148668b6ccc46846f673e (DEFAULT_TABLE table=test-table [id=c59090d4bc554d5696eacc4e9b450c1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:39:14.559469 7671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 975c91c2cf1148668b6ccc46846f673e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:39:14.559563 7398 tablet_service.cc:1467] Processing CreateTablet for tablet 975c91c2cf1148668b6ccc46846f673e (DEFAULT_TABLE table=test-table [id=c59090d4bc554d5696eacc4e9b450c1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:39:14.559806 7398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 975c91c2cf1148668b6ccc46846f673e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:39:14.561641 7839 tablet_bootstrap.cc:492] T 975c91c2cf1148668b6ccc46846f673e P 95c9bea2bd4044729172198bd0ee9bd4: Bootstrap starting.
I20260927 02:39:14.561666 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:14.554223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38069 (local address 127.0.0.1:48384)
0927 02:39:14.554486 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:14.554499 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:14.554583 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:14.555292 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:14.555295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:14.555310 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:39:14.555398 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:14.555403 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:14.556335 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:14.556339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:14.557569 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:14.557578 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:14.558492 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:14.558496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:14.558687 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:14.558713 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 02:39:14.558770 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 02:39:14.558834 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"spinlock_wait_cycles":2432,"thread_start_us":83,"threads_started":1}
I20260927 02:39:14.562162 7840 tablet_bootstrap.cc:492] T 975c91c2cf1148668b6ccc46846f673e P 6c73821393684ab0961970fdda44e818: Bootstrap starting.
I20260927 02:39:14.562212 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:14.554789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48384 (local address 127.0.98.2:38069)
0927 02:39:14.555068 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:14.555081 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:14.555104 (+ 23us) server_negotiation.cc:407] Connection header received
0927 02:39:14.555152 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:14.555155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:14.555198 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:14.555273 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:39:14.555489 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:14.556230 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:14.557704 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:14.558375 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:14.561907 (+ 3532us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:14.561968 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:14.562006 (+ 38us) server_negotiation.cc:299] Negotiation successful
0927 02:39:14.562088 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":63,"threads_started":1}
I20260927 02:39:14.562693 7517 tablet_service.cc:1467] Processing CreateTablet for tablet 975c91c2cf1148668b6ccc46846f673e (DEFAULT_TABLE table=test-table [id=c59090d4bc554d5696eacc4e9b450c1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:39:14.562927 7839 tablet_bootstrap.cc:654] T 975c91c2cf1148668b6ccc46846f673e P 95c9bea2bd4044729172198bd0ee9bd4: Neither blocks nor log segments found. Creating new log.
I20260927 02:39:14.562968 7517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 975c91c2cf1148668b6ccc46846f673e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:39:14.563062 7840 tablet_bootstrap.cc:654] T 975c91c2cf1148668b6ccc46846f673e P 6c73821393684ab0961970fdda44e818: Neither blocks nor log segments found. Creating new log.
I20260927 02:39:14.563974 7839 tablet_bootstrap.cc:492] T 975c91c2cf1148668b6ccc46846f673e P 95c9bea2bd4044729172198bd0ee9bd4: No bootstrap required, opened a new log
I20260927 02:39:14.564050 7839 ts_tablet_manager.cc:1397] T 975c91c2cf1148668b6ccc46846f673e P 95c9bea2bd4044729172198bd0ee9bd4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:39:14.564416 7839 raft_consensus.cc:348] T 975c91c2cf1148668b6ccc46846f673e P 95c9bea2bd4044729172198bd0ee9bd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95c9bea2bd4044729172198bd0ee9bd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44515 } } peers { permanent_uuid: "bfb91636c48348d1abc15549cd284922" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38069 } } peers { permanent_uuid: "6c73821393684ab0961970fdda44e818" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35281 } }
I20260927 02:39:14.564551 7839 raft_consensus.cc:374] T 975c91c2cf1148668b6ccc46846f673e P 95c9bea2bd4044729172198bd0ee9bd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:39:14.564591 7839 raft_consensus.cc:729] T 975c91c2cf1148668b6ccc46846f673e P 95c9bea2bd4044729172198bd0ee9bd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95c9bea2bd4044729172198bd0ee9bd4, State: Initialized, Role: FOLLOWER
I20260927 02:39:14.564714 7839 consensus_queue.cc:260] T 975c91c2cf1148668b6ccc46846f673e P 95c9bea2bd4044729172198bd0ee9bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95c9bea2bd4044729172198bd0ee9bd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44515 } } peers { permanent_uuid: "bfb91636c48348d1abc15549cd284922" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38069 } } peers { permanent_uuid: "6c73821393684ab0961970fdda44e818" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35281 } }
I20260927 02:39:14.565065 7839 ts_tablet_manager.cc:1428] T 975c91c2cf1148668b6ccc46846f673e P 95c9bea2bd4044729172198bd0ee9bd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:39:14.565235 7844 tablet_bootstrap.cc:492] T 975c91c2cf1148668b6ccc46846f673e P bfb91636c48348d1abc15549cd284922: Bootstrap starting.
I20260927 02:39:14.565913 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:14.565596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35857 (local address 127.0.98.3:60931)
0927 02:39:14.565796 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:14.565849 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260927 02:39:14.566113 7844 tablet_bootstrap.cc:654] T 975c91c2cf1148668b6ccc46846f673e P bfb91636c48348d1abc15549cd284922: Neither blocks nor log segments found. Creating new log.
I20260927 02:39:14.567139 7844 tablet_bootstrap.cc:492] T 975c91c2cf1148668b6ccc46846f673e P bfb91636c48348d1abc15549cd284922: No bootstrap required, opened a new log
I20260927 02:39:14.567210 7844 ts_tablet_manager.cc:1397] T 975c91c2cf1148668b6ccc46846f673e P bfb91636c48348d1abc15549cd284922: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:39:14.567603 7844 raft_consensus.cc:348] T 975c91c2cf1148668b6ccc46846f673e P bfb91636c48348d1abc15549cd284922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95c9bea2bd4044729172198bd0ee9bd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44515 } } peers { permanent_uuid: "bfb91636c48348d1abc15549cd284922" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38069 } } peers { permanent_uuid: "6c73821393684ab0961970fdda44e818" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35281 } }
I20260927 02:39:14.567723 7844 raft_consensus.cc:374] T 975c91c2cf1148668b6ccc46846f673e P bfb91636c48348d1abc15549cd284922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:39:14.567763 7844 raft_consensus.cc:729] T 975c91c2cf1148668b6ccc46846f673e P bfb91636c48348d1abc15549cd284922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfb91636c48348d1abc15549cd284922, State: Initialized, Role: FOLLOWER
I20260927 02:39:14.567889 7844 consensus_queue.cc:260] T 975c91c2cf1148668b6ccc46846f673e P bfb91636c48348d1abc15549cd284922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95c9bea2bd4044729172198bd0ee9bd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44515 } } peers { permanent_uuid: "bfb91636c48348d1abc15549cd284922" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38069 } } peers { permanent_uuid: "6c73821393684ab0961970fdda44e818" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35281 } }
I20260927 02:39:14.567983 7840 tablet_bootstrap.cc:492] T 975c91c2cf1148668b6ccc46846f673e P 6c73821393684ab0961970fdda44e818: No bootstrap required, opened a new log
I20260927 02:39:14.568068 7840 ts_tablet_manager.cc:1397] T 975c91c2cf1148668b6ccc46846f673e P 6c73821393684ab0961970fdda44e818: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260927 02:39:14.568240 7844 ts_tablet_manager.cc:1428] T 975c91c2cf1148668b6ccc46846f673e P bfb91636c48348d1abc15549cd284922: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:39:14.568532 7840 raft_consensus.cc:348] T 975c91c2cf1148668b6ccc46846f673e P 6c73821393684ab0961970fdda44e818 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95c9bea2bd4044729172198bd0ee9bd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44515 } } peers { permanent_uuid: "bfb91636c48348d1abc15549cd284922" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38069 } } peers { permanent_uuid: "6c73821393684ab0961970fdda44e818" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35281 } }
I20260927 02:39:14.568660 7840 raft_consensus.cc:374] T 975c91c2cf1148668b6ccc46846f673e P 6c73821393684ab0961970fdda44e818 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:39:14.568702 7840 raft_consensus.cc:729] T 975c91c2cf1148668b6ccc46846f673e P 6c73821393684ab0961970fdda44e818 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c73821393684ab0961970fdda44e818, State: Initialized, Role: FOLLOWER
I20260927 02:39:14.568835 7840 consensus_queue.cc:260] T 975c91c2cf1148668b6ccc46846f673e P 6c73821393684ab0961970fdda44e818 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95c9bea2bd4044729172198bd0ee9bd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44515 } } peers { permanent_uuid: "bfb91636c48348d1abc15549cd284922" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38069 } } peers { permanent_uuid: "6c73821393684ab0961970fdda44e818" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35281 } }
I20260927 02:39:14.568991 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:14.568585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35857 (local address 127.0.98.2:56641)
0927 02:39:14.568842 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:14.568927 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189}
I20260927 02:39:14.569188 7840 ts_tablet_manager.cc:1428] T 975c91c2cf1148668b6ccc46846f673e P 6c73821393684ab0961970fdda44e818: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260927 02:39:14.569947 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:14.569587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35857 (local address 127.0.98.1:47241)
0927 02:39:14.569816 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:14.569874 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
I20260927 02:39:14.571106 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:14.570832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35857 (local address 127.0.98.1:34477)
0927 02:39:14.571000 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:14.571049 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260927 02:39:14.571250 7466 heartbeater.cc:503] Master 127.0.98.61:40345 was elected leader, sending a full tablet report...
I20260927 02:39:14.571551 7603 heartbeater.cc:503] Master 127.0.98.61:40345 was elected leader, sending a full tablet report...
I20260927 02:39:14.571626 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:14.571377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35857 (local address 127.0.98.2:51731)
0927 02:39:14.571516 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:14.571563 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260927 02:39:14.647344 7846 raft_consensus.cc:482] T 975c91c2cf1148668b6ccc46846f673e P bfb91636c48348d1abc15549cd284922 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:39:14.647495 7846 raft_consensus.cc:504] T 975c91c2cf1148668b6ccc46846f673e P bfb91636c48348d1abc15549cd284922 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95c9bea2bd4044729172198bd0ee9bd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44515 } } peers { permanent_uuid: "bfb91636c48348d1abc15549cd284922" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38069 } } peers { permanent_uuid: "6c73821393684ab0961970fdda44e818" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35281 } }
I20260927 02:39:14.647897 7846 leader_election.cc:288] T 975c91c2cf1148668b6ccc46846f673e P bfb91636c48348d1abc15549cd284922 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 95c9bea2bd4044729172198bd0ee9bd4 (127.0.98.3:44515), 6c73821393684ab0961970fdda44e818 (127.0.98.1:35281)
I20260927 02:39:14.648375 7418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "975c91c2cf1148668b6ccc46846f673e" candidate_uuid: "bfb91636c48348d1abc15549cd284922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c73821393684ab0961970fdda44e818" is_pre_election: true
I20260927 02:39:14.648384 7691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "975c91c2cf1148668b6ccc46846f673e" candidate_uuid: "bfb91636c48348d1abc15549cd284922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95c9bea2bd4044729172198bd0ee9bd4" is_pre_election: true
I20260927 02:39:14.648530 7691 raft_consensus.cc:2454] T 975c91c2cf1148668b6ccc46846f673e P 95c9bea2bd4044729172198bd0ee9bd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfb91636c48348d1abc15549cd284922 in term 0.
I20260927 02:39:14.648546 7418 raft_consensus.cc:2454] T 975c91c2cf1148668b6ccc46846f673e P 6c73821393684ab0961970fdda44e818 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfb91636c48348d1abc15549cd284922 in term 0.
I20260927 02:39:14.648862 7488 leader_election.cc:302] T 975c91c2cf1148668b6ccc46846f673e P bfb91636c48348d1abc15549cd284922 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95c9bea2bd4044729172198bd0ee9bd4, bfb91636c48348d1abc15549cd284922; no voters:
I20260927 02:39:14.649061 7846 raft_consensus.cc:2781] T 975c91c2cf1148668b6ccc46846f673e P bfb91636c48348d1abc15549cd284922 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:39:14.649121 7846 raft_consensus.cc:482] T 975c91c2cf1148668b6ccc46846f673e P bfb91636c48348d1abc15549cd284922 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:39:14.649154 7846 raft_consensus.cc:3037] T 975c91c2cf1148668b6ccc46846f673e P bfb91636c48348d1abc15549cd284922 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:39:14.650377 7846 raft_consensus.cc:504] T 975c91c2cf1148668b6ccc46846f673e P bfb91636c48348d1abc15549cd284922 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95c9bea2bd4044729172198bd0ee9bd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44515 } } peers { permanent_uuid: "bfb91636c48348d1abc15549cd284922" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38069 } } peers { permanent_uuid: "6c73821393684ab0961970fdda44e818" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35281 } }
I20260927 02:39:14.650720 7846 leader_election.cc:288] T 975c91c2cf1148668b6ccc46846f673e P bfb91636c48348d1abc15549cd284922 [CANDIDATE]: Term 1 election: Requested vote from peers 95c9bea2bd4044729172198bd0ee9bd4 (127.0.98.3:44515), 6c73821393684ab0961970fdda44e818 (127.0.98.1:35281)
I20260927 02:39:14.651074 7691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "975c91c2cf1148668b6ccc46846f673e" candidate_uuid: "bfb91636c48348d1abc15549cd284922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95c9bea2bd4044729172198bd0ee9bd4"
I20260927 02:39:14.651095 7418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "975c91c2cf1148668b6ccc46846f673e" candidate_uuid: "bfb91636c48348d1abc15549cd284922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c73821393684ab0961970fdda44e818"
I20260927 02:39:14.651199 7691 raft_consensus.cc:3037] T 975c91c2cf1148668b6ccc46846f673e P 95c9bea2bd4044729172198bd0ee9bd4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:39:14.651198 7418 raft_consensus.cc:3037] T 975c91c2cf1148668b6ccc46846f673e P 6c73821393684ab0961970fdda44e818 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:39:14.652253 7691 raft_consensus.cc:2454] T 975c91c2cf1148668b6ccc46846f673e P 95c9bea2bd4044729172198bd0ee9bd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfb91636c48348d1abc15549cd284922 in term 1.
I20260927 02:39:14.652253 7418 raft_consensus.cc:2454] T 975c91c2cf1148668b6ccc46846f673e P 6c73821393684ab0961970fdda44e818 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfb91636c48348d1abc15549cd284922 in term 1.
I20260927 02:39:14.652521 7490 leader_election.cc:302] T 975c91c2cf1148668b6ccc46846f673e P bfb91636c48348d1abc15549cd284922 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c73821393684ab0961970fdda44e818, bfb91636c48348d1abc15549cd284922; no voters:
I20260927 02:39:14.652688 7846 raft_consensus.cc:2781] T 975c91c2cf1148668b6ccc46846f673e P bfb91636c48348d1abc15549cd284922 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:39:14.652906 7846 raft_consensus.cc:686] T 975c91c2cf1148668b6ccc46846f673e P bfb91636c48348d1abc15549cd284922 [term 1 LEADER]: Becoming Leader. State: Replica: bfb91636c48348d1abc15549cd284922, State: Running, Role: LEADER
I20260927 02:39:14.653045 7846 consensus_queue.cc:237] T 975c91c2cf1148668b6ccc46846f673e P bfb91636c48348d1abc15549cd284922 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95c9bea2bd4044729172198bd0ee9bd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44515 } } peers { permanent_uuid: "bfb91636c48348d1abc15549cd284922" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38069 } } peers { permanent_uuid: "6c73821393684ab0961970fdda44e818" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35281 } }
I20260927 02:39:14.653785 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:14.653354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35857 (local address 127.0.98.2:59449)
0927 02:39:14.653656 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:14.653714 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227}
I20260927 02:39:14.654898 7103 catalog_manager.cc:5495] T 975c91c2cf1148668b6ccc46846f673e P bfb91636c48348d1abc15549cd284922 reported cstate change: term changed from 0 to 1, leader changed from <none> to bfb91636c48348d1abc15549cd284922 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "bfb91636c48348d1abc15549cd284922" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95c9bea2bd4044729172198bd0ee9bd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfb91636c48348d1abc15549cd284922" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c73821393684ab0961970fdda44e818" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35281 } health_report { overall_health: UNKNOWN } } }
I20260927 02:39:14.680773 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:14.677885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48392 (local address 127.0.98.2:38069)
0927 02:39:14.678008 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:14.678012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:14.678029 (+ 17us) server_negotiation.cc:407] Connection header received
0927 02:39:14.678073 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:14.678075 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:14.678123 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:14.678184 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:39:14.678414 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:14.678909 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:14.679746 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:14.679957 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:14.680047 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:14.680356 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:39:14.680425 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:39:14.680478 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:14.680615 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 02:39:14.680662 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 02:39:14.682744 7418 raft_consensus.cc:1261] T 975c91c2cf1148668b6ccc46846f673e P 6c73821393684ab0961970fdda44e818 [term 1 FOLLOWER]: Refusing update from remote peer bfb91636c48348d1abc15549cd284922: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:39:14.682742 7691 raft_consensus.cc:1261] T 975c91c2cf1148668b6ccc46846f673e P 95c9bea2bd4044729172198bd0ee9bd4 [term 1 FOLLOWER]: Refusing update from remote peer bfb91636c48348d1abc15549cd284922: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:39:14.683244 7846 consensus_queue.cc:1035] T 975c91c2cf1148668b6ccc46846f673e P bfb91636c48348d1abc15549cd284922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95c9bea2bd4044729172198bd0ee9bd4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:39:14.683398 7848 consensus_queue.cc:1035] T 975c91c2cf1148668b6ccc46846f673e P bfb91636c48348d1abc15549cd284922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c73821393684ab0961970fdda44e818" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:39:14.684643 7739 heartbeater.cc:503] Master 127.0.98.61:40345 was elected leader, sending a full tablet report...
I20260927 02:39:14.684960 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:14.684507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35857 (local address 127.0.98.1:51379)
0927 02:39:14.684830 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:14.684890 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257}
I20260927 02:39:14.685567 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:14.685306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35857 (local address 127.0.98.3:39349)
0927 02:39:14.685449 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:14.685502 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 02:39:14.687695 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:14.687379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35857 (local address 127.0.98.2:54283)
0927 02:39:14.687578 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:14.687629 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
I20260927 02:39:14.688114 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:14.687833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35857 (local address 127.0.98.1:42703)
0927 02:39:14.688009 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:14.688051 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260927 02:39:14.690114 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:14.689704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35857 (local address 127.0.98.3:58871)
0927 02:39:14.689993 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:14.690049 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226}
I20260927 02:39:14.690653 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:14.690375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35857 (local address 127.0.98.2:52321)
0927 02:39:14.690547 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:14.690594 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260927 02:39:14.693204 7103 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:39:14.693328 7103 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:39:14.696537 7103 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6f4dad5a3ce0416eb97e8a4be06dff38: Sending DeleteTablet for 3 replicas of tablet 975c91c2cf1148668b6ccc46846f673e
I20260927 02:39:14.697085 7671 tablet_service.cc:1514] Processing DeleteTablet for tablet 975c91c2cf1148668b6ccc46846f673e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:39:14 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:49084
I20260927 02:39:14.697424 7517 tablet_service.cc:1514] Processing DeleteTablet for tablet 975c91c2cf1148668b6ccc46846f673e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:39:14 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:48384
I20260927 02:39:14.697583 7398 tablet_service.cc:1514] Processing DeleteTablet for tablet 975c91c2cf1148668b6ccc46846f673e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:39:14 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:39426
I20260927 02:39:14.697741 7860 tablet_replica.cc:330] T 975c91c2cf1148668b6ccc46846f673e P 95c9bea2bd4044729172198bd0ee9bd4: stopping tablet replica
I20260927 02:39:14.697758 7861 tablet_replica.cc:330] T 975c91c2cf1148668b6ccc46846f673e P bfb91636c48348d1abc15549cd284922: stopping tablet replica
I20260927 02:39:14.697865 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 7247
I20260927 02:39:14.697857 7860 raft_consensus.cc:2229] T 975c91c2cf1148668b6ccc46846f673e P 95c9bea2bd4044729172198bd0ee9bd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:39:14.697886 7862 tablet_replica.cc:330] T 975c91c2cf1148668b6ccc46846f673e P 6c73821393684ab0961970fdda44e818: stopping tablet replica
I20260927 02:39:14.697978 7860 raft_consensus.cc:2258] T 975c91c2cf1148668b6ccc46846f673e P 95c9bea2bd4044729172198bd0ee9bd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:39:14.698161 7861 raft_consensus.cc:2229] T 975c91c2cf1148668b6ccc46846f673e P bfb91636c48348d1abc15549cd284922 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:39:14.698581 7861 raft_consensus.cc:2258] T 975c91c2cf1148668b6ccc46846f673e P bfb91636c48348d1abc15549cd284922 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:39:14.699208 7861 ts_tablet_manager.cc:1905] T 975c91c2cf1148668b6ccc46846f673e P bfb91636c48348d1abc15549cd284922: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:39:14.699738 7860 ts_tablet_manager.cc:1905] T 975c91c2cf1148668b6ccc46846f673e P 95c9bea2bd4044729172198bd0ee9bd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:39:14.701591 7860 ts_tablet_manager.cc:1918] T 975c91c2cf1148668b6ccc46846f673e P 95c9bea2bd4044729172198bd0ee9bd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:39:14.701658 7860 log.cc:1196] T 975c91c2cf1148668b6ccc46846f673e P 95c9bea2bd4044729172198bd0ee9bd4: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-2/wal/wals/975c91c2cf1148668b6ccc46846f673e
I20260927 02:39:14.701666 7861 ts_tablet_manager.cc:1918] T 975c91c2cf1148668b6ccc46846f673e P bfb91636c48348d1abc15549cd284922: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:39:14.701730 7861 log.cc:1196] T 975c91c2cf1148668b6ccc46846f673e P bfb91636c48348d1abc15549cd284922: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790476718023131-392-0/minicluster-data/ts-1/wal/wals/975c91c2cf1148668b6ccc46846f673e
I20260927 02:39:14.701882 7860 ts_tablet_manager.cc:1939] T 975c91c2cf1148668b6ccc46846f673e P 95c9bea2bd4044729172198bd0ee9bd4: Deleting consensus metadata
I20260927 02:39:14.701985 7861 ts_tablet_manager.cc:1939] T 975c91c2cf1148668b6ccc46846f673e P bfb91636c48348d1abc15549cd284922: Deleting consensus metadata
I20260927 02:39:14.702558 7087 catalog_manager.cc:4841] TS 95c9bea2bd4044729172198bd0ee9bd4 (127.0.98.3:44515): tablet 975c91c2cf1148668b6ccc46846f673e (table test-table [id=c59090d4bc554d5696eacc4e9b450c1c]) successfully deleted
I20260927 02:39:14.702940 7088 catalog_manager.cc:4841] TS bfb91636c48348d1abc15549cd284922 (127.0.98.2:38069): tablet 975c91c2cf1148668b6ccc46846f673e (table test-table [id=c59090d4bc554d5696eacc4e9b450c1c]) successfully deleted
I20260927 02:39:14.705400 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 7472
I20260927 02:39:14.705626 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:14.705383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35281 (local address 127.0.0.1:39434)
0927 02:39:14.705507 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:14.705556 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 02:39:14.706094 7089 catalog_manager.cc:4584] TS 6c73821393684ab0961970fdda44e818 (127.0.98.1:35281): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 975c91c2cf1148668b6ccc46846f673e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35281: connect: Connection refused (error 111)
I20260927 02:39:14.711664 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 7608
I20260927 02:39:14.716100 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 6966
I20260927 02:39:14.720680 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 7073
2026-09-27T02:39:14Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6983 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:39:14 dist-test-slave-zv8z krb5kdc[7865](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:39:14 dist-test-slave-zv8z krb5kdc[7865](info): set up 2 sockets
Sep 27 02:39:14 dist-test-slave-zv8z krb5kdc[7865](info): commencing operation
krb5kdc: starting...
W20260927 02:39:15.678609 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/krb5kdc/test-user.keytab.
Sep 27 02:39:15 dist-test-slave-zv8z krb5kdc[7865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:39:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:39:15Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260927 02:39:15.793421 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37181
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:39:15.900311 7878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:39:15.900624 7878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:39:15.900669 7878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:39:15.904712 7878 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:39:15.904770 7878 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 02:39:15.904785 7878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:39:15.904801 7878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:39:15.904816 7878 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:39:15.904829 7878 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:39:15.909994 7878 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37181
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:39:15.910987 7878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:39:15.911933 7878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:39:15.920440 7878 server_base.cc:1034] running on GCE node
W20260927 02:39:15.920408 7883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:15.920401 7884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:15.920399 7886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:15.921178 7878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:39:15.922214 7878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:39:15.923471 7878 hybrid_clock.cc:648] HybridClock initialized: now 1790476755923452 us; error 35 us; skew 500 ppm
Sep 27 02:39:15 dist-test-slave-zv8z krb5kdc[7865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476755, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:39:15.926107 7878 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260927 02:39:15.927042 7878 webserver.cc:460] Webserver started at http://127.0.98.62:36457/ using document root <none> and password file <none>
I20260927 02:39:15.927567 7878 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:15.927631 7878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:15.927822 7878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:39:15.930022 7878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/master-0/data/instance:
uuid: "6c03c951086c4af5b892870108fd8990"
format_stamp: "Formatted at 2026-09-27 02:39:15 on dist-test-slave-zv8z"
server_key: "643f3a04eb8898171fa665cef154a321"
server_key_iv: "b56063e9d71230543d3a751ac92d72d8"
server_key_version: "encryptionkey@0"
I20260927 02:39:15.930593 7878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/master-0/wal/instance:
uuid: "6c03c951086c4af5b892870108fd8990"
format_stamp: "Formatted at 2026-09-27 02:39:15 on dist-test-slave-zv8z"
server_key: "643f3a04eb8898171fa665cef154a321"
server_key_iv: "b56063e9d71230543d3a751ac92d72d8"
server_key_version: "encryptionkey@0"
I20260927 02:39:15.934182 7878 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 02:39:15.936597 7893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:15.938122 7878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 02:39:15.938230 7878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "6c03c951086c4af5b892870108fd8990"
format_stamp: "Formatted at 2026-09-27 02:39:15 on dist-test-slave-zv8z"
server_key: "643f3a04eb8898171fa665cef154a321"
server_key_iv: "b56063e9d71230543d3a751ac92d72d8"
server_key_version: "encryptionkey@0"
I20260927 02:39:15.938370 7878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:15.962020 7878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:39:15.964958 7878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:39:15.965131 7878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:39:15.972436 7878 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37181
I20260927 02:39:15.972461 7955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37181 every 8 connection(s)
I20260927 02:39:15.973359 7878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:39:15.976028 7956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:39:15.979144 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 7878
I20260927 02:39:15.979261 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/master-0/wal/instance
I20260927 02:39:15.979558 392 external_mini_cluster.cc:1428] Setting key 4e15102ec1a2b23d358c4fe4db7e890b
I20260927 02:39:15.981887 7956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c03c951086c4af5b892870108fd8990: Bootstrap starting.
I20260927 02:39:15.984022 7956 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c03c951086c4af5b892870108fd8990: Neither blocks nor log segments found. Creating new log.
I20260927 02:39:15.984746 7956 log.cc:824] T 00000000000000000000000000000000 P 6c03c951086c4af5b892870108fd8990: Log is configured to *not* fsync() on all Append() calls
I20260927 02:39:15.986590 7956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c03c951086c4af5b892870108fd8990: No bootstrap required, opened a new log
Sep 27 02:39:15 dist-test-slave-zv8z krb5kdc[7865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:39:15.989889 7956 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c03c951086c4af5b892870108fd8990 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c03c951086c4af5b892870108fd8990" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37181 } }
I20260927 02:39:15.990108 7956 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c03c951086c4af5b892870108fd8990 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:39:15.990144 7956 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c03c951086c4af5b892870108fd8990 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c03c951086c4af5b892870108fd8990, State: Initialized, Role: FOLLOWER
I20260927 02:39:15.990587 7956 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c03c951086c4af5b892870108fd8990 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c03c951086c4af5b892870108fd8990" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37181 } }
I20260927 02:39:15.990712 7956 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c03c951086c4af5b892870108fd8990 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:39:15.990773 7956 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c03c951086c4af5b892870108fd8990 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:39:15.990857 7956 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c03c951086c4af5b892870108fd8990 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:39:15.991722 7956 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c03c951086c4af5b892870108fd8990 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c03c951086c4af5b892870108fd8990" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37181 } }
I20260927 02:39:15.992022 7956 leader_election.cc:302] T 00000000000000000000000000000000 P 6c03c951086c4af5b892870108fd8990 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c03c951086c4af5b892870108fd8990; no voters:
I20260927 02:39:15.992307 7956 leader_election.cc:288] T 00000000000000000000000000000000 P 6c03c951086c4af5b892870108fd8990 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:39:15.992376 7961 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c03c951086c4af5b892870108fd8990 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:39:15.992554 7961 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c03c951086c4af5b892870108fd8990 [term 1 LEADER]: Becoming Leader. State: Replica: 6c03c951086c4af5b892870108fd8990, State: Running, Role: LEADER
I20260927 02:39:15.992848 7961 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c03c951086c4af5b892870108fd8990 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c03c951086c4af5b892870108fd8990" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37181 } }
I20260927 02:39:15.993266 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:15.981058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35692 (local address 127.0.98.62:37181)
0927 02:39:15.981478 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:15.981493 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:15.981529 (+ 36us) server_negotiation.cc:407] Connection header received
0927 02:39:15.982173 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:15.982191 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:15.982538 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:15.982835 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:15.983602 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:15.984673 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:15.985654 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:15.986100 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:15.986275 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:15.988607 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:15.988612 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:15.988624 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:15.988649 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:15.990380 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:15.990780 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:15.990786 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:15.990792 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:15.990925 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:15.991176 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:15.991179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:15.991181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:15.991613 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:15.991798 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:15.992049 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 02:39:15.992365 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":125,"threads_started":1}
I20260927 02:39:15.993548 7956 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c03c951086c4af5b892870108fd8990 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:39:15.995847 7963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c03c951086c4af5b892870108fd8990 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c03c951086c4af5b892870108fd8990. Latest consensus state: current_term: 1 leader_uuid: "6c03c951086c4af5b892870108fd8990" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c03c951086c4af5b892870108fd8990" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37181 } } }
I20260927 02:39:15.995968 7963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c03c951086c4af5b892870108fd8990 [sys.catalog]: This master's current role is: LEADER
I20260927 02:39:15.995996 7962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c03c951086c4af5b892870108fd8990 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c03c951086c4af5b892870108fd8990" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c03c951086c4af5b892870108fd8990" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37181 } } }
I20260927 02:39:15.996093 7962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c03c951086c4af5b892870108fd8990 [sys.catalog]: This master's current role is: LEADER
W20260927 02:39:15.998503 7975 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6c03c951086c4af5b892870108fd8990: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:39:15.998566 7975 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:39:15.998651 7976 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:39:15.999347 7976 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:39:16.004957 7976 catalog_manager.cc:1347] Generated new cluster ID: 04043f7886c24c2db14b0451c763509f
I20260927 02:39:16.005026 7976 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:39:16.012034 7976 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:39:16.013259 7976 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:39:16.025143 7976 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c03c951086c4af5b892870108fd8990: Generated new TSK 0
I20260927 02:39:16.025974 7976 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260927 02:39:16.064771 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37181
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:39:16.185259 7982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:39:16.185534 7982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:39:16.185577 7982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:39:16.189535 7982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:39:16.189600 7982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:39:16.189638 7982 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:39:16.189669 7982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:39:16.195012 7982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:37181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:39:16.196063 7982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:39:16.197031 7982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:39:16.206277 7990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:16.206296 7987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:16.206399 7988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:16.206650 7982 server_base.cc:1034] running on GCE node
I20260927 02:39:16.207053 7982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:39:16.207636 7982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:39:16.208809 7982 hybrid_clock.cc:648] HybridClock initialized: now 1790476756208789 us; error 35 us; skew 500 ppm
Sep 27 02:39:16 dist-test-slave-zv8z krb5kdc[7865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476756, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:39:16.211565 7982 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 02:39:16.212540 7982 webserver.cc:460] Webserver started at http://127.0.98.1:33065/ using document root <none> and password file <none>
I20260927 02:39:16.213074 7982 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:16.213131 7982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:16.213294 7982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:39:16.215405 7982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-0/data/instance:
uuid: "d584780e230c4de9b74928bec112ad9a"
format_stamp: "Formatted at 2026-09-27 02:39:16 on dist-test-slave-zv8z"
server_key: "468baab58b4cdcf2f399489d00337a4e"
server_key_iv: "b035aeb294096dd99c6377a2e25846a3"
server_key_version: "encryptionkey@0"
I20260927 02:39:16.215926 7982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d584780e230c4de9b74928bec112ad9a"
format_stamp: "Formatted at 2026-09-27 02:39:16 on dist-test-slave-zv8z"
server_key: "468baab58b4cdcf2f399489d00337a4e"
server_key_iv: "b035aeb294096dd99c6377a2e25846a3"
server_key_version: "encryptionkey@0"
I20260927 02:39:16.219550 7982 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:39:16.221935 7997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:16.223219 7982 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:39:16.223318 7982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-0/wal
uuid: "d584780e230c4de9b74928bec112ad9a"
format_stamp: "Formatted at 2026-09-27 02:39:16 on dist-test-slave-zv8z"
server_key: "468baab58b4cdcf2f399489d00337a4e"
server_key_iv: "b035aeb294096dd99c6377a2e25846a3"
server_key_version: "encryptionkey@0"
I20260927 02:39:16.223428 7982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:16.243884 7982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:39:16.247196 7982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:39:16.247367 7982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:39:16.248817 7982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:39:16.248878 7982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:16.248930 7982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:39:16.248960 7982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 02:39:16 dist-test-slave-zv8z krb5kdc[7865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476756, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:39:16.258945 7982 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44417
I20260927 02:39:16.258975 8112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44417 every 8 connection(s)
I20260927 02:39:16.259799 7982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:39:16.261523 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 7982
I20260927 02:39:16.261647 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:39:16.261989 392 external_mini_cluster.cc:1428] Setting key 6ca1809fa166f6d8d9b362b72a195064
I20260927 02:39:16.262905 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:16.249450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40001 (local address 127.0.98.62:37181)
0927 02:39:16.249587 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:16.249592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:16.250540 (+ 948us) server_negotiation.cc:407] Connection header received
0927 02:39:16.251717 (+ 1177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:16.251721 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:16.251780 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:16.251839 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:16.253583 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:16.254313 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:16.256009 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:16.256327 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:16.256422 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:16.259252 (+ 2830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:16.259255 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:16.259258 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:16.259361 (+ 103us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:16.260923 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:16.261391 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:16.261394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:16.261396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:16.261485 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:16.261849 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:16.261852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:16.261854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:16.262082 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:16.262179 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:16.262650 (+ 471us) server_negotiation.cc:299] Negotiation successful
0927 02:39:16.262786 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 02:39:16.263832 8006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:16.249855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37181 (local address 127.0.98.1:40001)
0927 02:39:16.250372 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:16.250432 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:16.251458 (+ 1026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:16.252263 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:16.252272 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:16.252736 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:16.253414 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:16.253434 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:16.254447 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:16.254451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:16.255874 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:16.255886 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:16.256482 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:16.256486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:16.256690 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:16.257370 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:16.257393 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:16.259092 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:16.261017 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:16.261021 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:16.261032 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:16.261295 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:16.261570 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:16.261572 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:16.261574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:16.261737 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:16.262170 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:16.262175 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:16.262451 (+ 276us) client_negotiation.cc:769] Sending connection context
0927 02:39:16.262605 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 02:39:16.262925 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":115,"threads_started":1}
I20260927 02:39:16.265085 8113 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37181
I20260927 02:39:16.265224 8113 heartbeater.cc:464] Registering TS with master...
I20260927 02:39:16.265681 8113 heartbeater.cc:511] Master 127.0.98.62:37181 requested a full tablet report, sending...
I20260927 02:39:16.267213 7909 ts_manager.cc:194] Registered new tserver with Master: d584780e230c4de9b74928bec112ad9a (127.0.98.1:44417)
I20260927 02:39:16.268782 7909 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40001
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260927 02:39:16.274230 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:16.267267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52293 (local address 127.0.98.62:37181)
0927 02:39:16.267368 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:16.267371 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:16.267583 (+ 212us) server_negotiation.cc:407] Connection header received
0927 02:39:16.267723 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:16.267726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:16.267768 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:16.267834 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:16.268237 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:16.268730 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:16.269557 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:16.269895 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:16.269993 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:16.271307 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:16.271360 (+ 53us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:16.271362 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:16.271387 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:16.272752 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:16.273116 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:16.273118 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:16.273119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:16.273185 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:16.273489 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:16.273493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:16.273495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:16.273675 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:16.273786 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:16.273982 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 02:39:16.274077 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 02:39:16.274362 8120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:16.267205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37181 (local address 127.0.98.1:52293)
0927 02:39:16.267508 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:16.267521 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:16.267647 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:16.267858 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:16.267861 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:16.268045 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:16.268136 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:16.268140 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:16.268840 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:16.268843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:16.269461 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:16.269471 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:16.270112 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:16.270116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:16.270212 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:16.270730 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:16.270744 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:16.271148 (+ 404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:16.272859 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:16.272861 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:16.272863 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:16.273044 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:16.273252 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:16.273254 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:16.273255 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:16.273383 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:16.273777 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:16.273779 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:16.273853 (+ 74us) client_negotiation.cc:769] Sending connection context
0927 02:39:16.273935 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 02:39:16.274074 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":105,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260927 02:39:16.289873 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37181
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:39:16.396953 8122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:39:16.397230 8122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:39:16.397294 8122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:39:16.401407 8122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:39:16.401468 8122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:39:16.401533 8122 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:39:16.401587 8122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:39:16.406797 8122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:37181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:39:16.407795 8122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:39:16.408783 8122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:39:16.417858 8130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:16.417851 8127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:16.417861 8128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:16.418181 8122 server_base.cc:1034] running on GCE node
I20260927 02:39:16.418572 8122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:39:16.419158 8122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:39:16.420322 8122 hybrid_clock.cc:648] HybridClock initialized: now 1790476756420304 us; error 29 us; skew 500 ppm
Sep 27 02:39:16 dist-test-slave-zv8z krb5kdc[7865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476756, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:39:16.422866 8122 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 02:39:16.423730 8122 webserver.cc:460] Webserver started at http://127.0.98.2:33627/ using document root <none> and password file <none>
I20260927 02:39:16.424229 8122 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:16.424290 8122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:16.424469 8122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:39:16.426405 8122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-1/data/instance:
uuid: "6feca25dab664c3ebcd5e224abf3e8a7"
format_stamp: "Formatted at 2026-09-27 02:39:16 on dist-test-slave-zv8z"
server_key: "d4a6f206b66fcca6d5b837fd7b3d9812"
server_key_iv: "fd4f73a29c5782d61331df9fae7bc20c"
server_key_version: "encryptionkey@0"
I20260927 02:39:16.426925 8122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6feca25dab664c3ebcd5e224abf3e8a7"
format_stamp: "Formatted at 2026-09-27 02:39:16 on dist-test-slave-zv8z"
server_key: "d4a6f206b66fcca6d5b837fd7b3d9812"
server_key_iv: "fd4f73a29c5782d61331df9fae7bc20c"
server_key_version: "encryptionkey@0"
I20260927 02:39:16.430451 8122 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:39:16.432893 8137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:16.434063 8122 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:39:16.434165 8122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-1/wal
uuid: "6feca25dab664c3ebcd5e224abf3e8a7"
format_stamp: "Formatted at 2026-09-27 02:39:16 on dist-test-slave-zv8z"
server_key: "d4a6f206b66fcca6d5b837fd7b3d9812"
server_key_iv: "fd4f73a29c5782d61331df9fae7bc20c"
server_key_version: "encryptionkey@0"
I20260927 02:39:16.434278 8122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:16.454459 8122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:39:16.457402 8122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:39:16.457583 8122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:39:16.459022 8122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:39:16.459071 8122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:16.459123 8122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:39:16.459151 8122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 02:39:16 dist-test-slave-zv8z krb5kdc[7865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476756, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:39:16.468871 8122 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39041
I20260927 02:39:16.468982 8252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39041 every 8 connection(s)
I20260927 02:39:16.469892 8122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:39:16.471643 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:16.459782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36477 (local address 127.0.98.62:37181)
0927 02:39:16.459933 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:16.459939 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:16.460767 (+ 828us) server_negotiation.cc:407] Connection header received
0927 02:39:16.461954 (+ 1187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:16.461958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:16.462016 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:16.462104 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:16.463299 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:16.463817 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:16.465177 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:16.465446 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:16.465534 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:16.468203 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:16.468206 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:16.468209 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:16.468237 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:16.469794 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:16.470328 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:16.470331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:16.470333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:16.470409 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:16.470752 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:16.470755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:16.470756 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:16.470956 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:16.471031 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:16.471444 (+ 413us) server_negotiation.cc:299] Negotiation successful
0927 02:39:16.471530 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 02:39:16.472507 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:16.460119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37181 (local address 127.0.98.2:36477)
0927 02:39:16.460594 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:16.460652 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:16.461659 (+ 1007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:16.462364 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:16.462372 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:16.462678 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:16.463163 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:16.463176 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:16.463927 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:16.463930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:16.465062 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:16.465071 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:16.465613 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:16.465618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:16.465805 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:16.466398 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:16.466413 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:16.468054 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:16.469906 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:16.469911 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:16.469922 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:16.470180 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:16.470487 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:16.470490 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:16.470492 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:16.470660 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:16.471039 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:16.471044 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:16.471262 (+ 218us) client_negotiation.cc:769] Sending connection context
0927 02:39:16.471412 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 02:39:16.471635 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":110,"threads_started":1}
I20260927 02:39:16.473564 8253 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37181
I20260927 02:39:16.473687 8253 heartbeater.cc:464] Registering TS with master...
I20260927 02:39:16.474116 8253 heartbeater.cc:511] Master 127.0.98.62:37181 requested a full tablet report, sending...
I20260927 02:39:16.475154 7910 ts_manager.cc:194] Registered new tserver with Master: 6feca25dab664c3ebcd5e224abf3e8a7 (127.0.98.2:39041)
I20260927 02:39:16.475853 7910 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36477
I20260927 02:39:16.475927 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 8122
I20260927 02:39:16.476056 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:39:16.476374 392 external_mini_cluster.cc:1428] Setting key fe8cd82c9c45e68cff921dd75117b238
I20260927 02:39:16.484252 8259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:16.475489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37181 (local address 127.0.98.2:46277)
0927 02:39:16.475853 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:16.475867 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:16.475961 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:16.476243 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:16.476246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:16.477813 (+ 1567us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:16.477902 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:16.477907 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:16.478845 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:16.478848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:16.479657 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:16.479668 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:16.480070 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:16.480073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:16.480160 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:16.480604 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:16.480617 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:16.480999 (+ 382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:16.482800 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:16.482803 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:16.482805 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:16.482993 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:16.483288 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:16.483291 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:16.483292 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:16.483387 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:16.483837 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:16.483840 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:16.483917 (+ 77us) client_negotiation.cc:769] Sending connection context
0927 02:39:16.484000 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 02:39:16.484105 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":95,"threads_started":1}
I20260927 02:39:16.484325 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:16.475580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46277 (local address 127.0.98.62:37181)
0927 02:39:16.475701 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:16.475705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:16.475947 (+ 242us) server_negotiation.cc:407] Connection header received
0927 02:39:16.476088 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:16.476091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:16.476137 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:16.476217 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:16.478025 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:16.478738 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:16.479766 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:16.479970 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:16.480044 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:16.481133 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:16.481136 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:16.481138 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:16.481159 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:16.482698 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:16.483075 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:16.483079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:16.483080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:16.483165 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:16.483474 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:16.483477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:16.483479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:16.483737 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:16.483842 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:16.484068 (+ 226us) server_negotiation.cc:299] Negotiation successful
0927 02:39:16.484177 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 02:39:16.501588 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37181
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:39:16.606703 8262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:39:16.607052 8262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:39:16.607151 8262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:39:16.611191 8262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:39:16.611244 8262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:39:16.611296 8262 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:39:16.611351 8262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:39:16.616434 8262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:37181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:39:16.617362 8262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:39:16.618314 8262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:39:16.627164 8268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:16.627333 8262 server_base.cc:1034] running on GCE node
W20260927 02:39:16.627159 8267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:16.627178 8270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:16.627781 8262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:39:16.628376 8262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:39:16.629550 8262 hybrid_clock.cc:648] HybridClock initialized: now 1790476756629536 us; error 24 us; skew 500 ppm
Sep 27 02:39:16 dist-test-slave-zv8z krb5kdc[7865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476756, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:39:16.632089 8262 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 02:39:16.632846 8262 webserver.cc:460] Webserver started at http://127.0.98.3:43563/ using document root <none> and password file <none>
I20260927 02:39:16.633320 8262 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:16.633373 8262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:16.633526 8262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:39:16.635473 8262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-2/data/instance:
uuid: "97e535c7163e495a87620630469ab388"
format_stamp: "Formatted at 2026-09-27 02:39:16 on dist-test-slave-zv8z"
server_key: "1b8342b2f8863c0e11af1ea65eae3d16"
server_key_iv: "3446f3b2ee71f0875b8be3baf7f142be"
server_key_version: "encryptionkey@0"
I20260927 02:39:16.635959 8262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-2/wal/instance:
uuid: "97e535c7163e495a87620630469ab388"
format_stamp: "Formatted at 2026-09-27 02:39:16 on dist-test-slave-zv8z"
server_key: "1b8342b2f8863c0e11af1ea65eae3d16"
server_key_iv: "3446f3b2ee71f0875b8be3baf7f142be"
server_key_version: "encryptionkey@0"
I20260927 02:39:16.639365 8262 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:39:16.641610 8277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:16.642820 8262 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:39:16.642921 8262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-2/wal
uuid: "97e535c7163e495a87620630469ab388"
format_stamp: "Formatted at 2026-09-27 02:39:16 on dist-test-slave-zv8z"
server_key: "1b8342b2f8863c0e11af1ea65eae3d16"
server_key_iv: "3446f3b2ee71f0875b8be3baf7f142be"
server_key_version: "encryptionkey@0"
I20260927 02:39:16.643007 8262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:16.681092 8262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:39:16.684016 8262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:39:16.684181 8262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:39:16.685535 8262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:39:16.685581 8262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:16.685632 8262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:39:16.685659 8262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 02:39:16 dist-test-slave-zv8z krb5kdc[7865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476756, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:39:16.695456 8262 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41159
I20260927 02:39:16.695547 8392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41159 every 8 connection(s)
I20260927 02:39:16.696417 8262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:39:16.697541 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 8262
I20260927 02:39:16.697669 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:39:16.697960 392 external_mini_cluster.cc:1428] Setting key 31a96898d2ac16243b85348c7484173c
I20260927 02:39:16.698463 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:16.686396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45835 (local address 127.0.98.62:37181)
0927 02:39:16.686531 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:16.686536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:16.687402 (+ 866us) server_negotiation.cc:407] Connection header received
0927 02:39:16.688732 (+ 1330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:16.688736 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:16.688792 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:16.688885 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:16.690121 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:16.690777 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:16.691951 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:16.692166 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:16.692249 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:16.694903 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:16.694906 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:16.694909 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:16.694939 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:16.696390 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:16.696948 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:16.696951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:16.696953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:16.697040 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:16.697385 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:16.697388 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:16.697390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:16.697637 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:16.697737 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:16.698193 (+ 456us) server_negotiation.cc:299] Negotiation successful
0927 02:39:16.698333 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 02:39:16.699200 8286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:16.686772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37181 (local address 127.0.98.3:45835)
0927 02:39:16.687236 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:16.687295 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:16.688503 (+ 1208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:16.689068 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:16.689074 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:16.689448 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:16.689986 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:16.689999 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:16.690894 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:16.690898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:16.691831 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:16.691840 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:16.692311 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:16.692315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:16.692496 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:16.693046 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:16.693067 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:16.694745 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:16.696503 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:16.696508 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:16.696521 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:16.696838 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:16.697137 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:16.697140 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:16.697142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:16.697289 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:16.697736 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:16.697742 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:16.697964 (+ 222us) client_negotiation.cc:769] Sending connection context
0927 02:39:16.698128 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 02:39:16.698420 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":110,"threads_started":1}
I20260927 02:39:16.700206 8393 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37181
I20260927 02:39:16.700397 8393 heartbeater.cc:464] Registering TS with master...
I20260927 02:39:16.700819 8393 heartbeater.cc:511] Master 127.0.98.62:37181 requested a full tablet report, sending...
I20260927 02:39:16.701713 7910 ts_manager.cc:194] Registered new tserver with Master: 97e535c7163e495a87620630469ab388 (127.0.98.3:41159)
I20260927 02:39:16.702409 7910 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45835
I20260927 02:39:16.707942 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:16.702150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40939 (local address 127.0.98.62:37181)
0927 02:39:16.702280 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:16.702284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:16.702423 (+ 139us) server_negotiation.cc:407] Connection header received
0927 02:39:16.702513 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:16.702516 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:16.702553 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:16.702615 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:16.702989 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:16.703468 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:16.704232 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:16.704450 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:16.704529 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:16.705382 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:16.705385 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:16.705388 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:16.705409 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:16.706529 (+ 1120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:16.706886 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:16.706889 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:16.706891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:16.706966 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:16.707193 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:16.707196 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:16.707198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:16.707349 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:16.707480 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:16.707713 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 02:39:16.707803 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 02:39:16.707955 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:16.702103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37181 (local address 127.0.98.3:40939)
0927 02:39:16.702355 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:16.702367 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:16.702447 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:16.702634 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:16.702637 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:16.702812 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:16.702895 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:16.702899 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:16.703576 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:16.703579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:16.704135 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:16.704142 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:16.704539 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:16.704543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:16.704631 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:16.704999 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:16.705009 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:16.705267 (+ 258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:16.706624 (+ 1357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:16.706628 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:16.706630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:16.706804 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:16.707041 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:16.707044 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:16.707046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:16.707118 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:16.707480 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:16.707482 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:16.707568 (+ 86us) client_negotiation.cc:769] Sending connection context
0927 02:39:16.707652 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 02:39:16.707776 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":89,"threads_started":1}
I20260927 02:39:16.711211 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 02:39:16 dist-test-slave-zv8z krb5kdc[7865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476756, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 02:39:16 dist-test-slave-zv8z krb5kdc[7865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:16 dist-test-slave-zv8z krb5kdc[7865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 02:39:16.732285 7959 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:16.726855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35694 (local address 127.0.98.62:37181)
0927 02:39:16.726963 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:16.726966 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:16.726977 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:39:16.727011 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:16.727013 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:16.727046 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:16.727109 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:16.727782 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:16.728335 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:16.729143 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:16.729345 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:16.729434 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:16.732147 (+ 2713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35694: BlockingRecv error: recv got EOF from 127.0.0.1:35694 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 27 02:39:16 dist-test-slave-zv8z krb5kdc[7865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476756, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:39:16.740698 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:16.733560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35698 (local address 127.0.98.62:37181)
0927 02:39:16.733684 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:16.733687 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:16.733766 (+ 79us) server_negotiation.cc:407] Connection header received
0927 02:39:16.733852 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:16.733854 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:16.733886 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:16.733944 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:16.734501 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:16.734967 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:16.735670 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:16.735867 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:16.735939 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:16.737933 (+ 1994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:16.737936 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:16.737938 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:16.737958 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:16.739424 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:16.739774 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:16.739776 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:16.739777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:16.739840 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:16.740023 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:16.740025 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:16.740026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:16.740163 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:16.740263 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:16.740477 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 02:39:16.740554 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 02:39:16.743404 7910 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35698:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:39:16.746333 7910 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:39:16.760149 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:16.754314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45594 (local address 127.0.98.2:39041)
0927 02:39:16.756068 (+ 1754us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:16.756076 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:16.756092 (+ 16us) server_negotiation.cc:407] Connection header received
0927 02:39:16.756148 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:16.756152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:16.756339 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:16.756485 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:39:16.756598 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:16.757622 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:16.758977 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:16.759503 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:16.759597 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:16.759754 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:16.759899 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 02:39:16.759995 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1673,"thread_start_us":101,"threads_started":1}
I20260927 02:39:16.760150 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:16.754137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39041 (local address 127.0.0.1:45594)
0927 02:39:16.754872 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:16.754932 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:16.755047 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:16.756460 (+ 1413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:16.756462 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:16.756476 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:39:16.756538 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:16.756542 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:16.757738 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:16.757741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:16.758878 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:16.758886 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:16.759610 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:16.759614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:16.759713 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:16.759748 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 02:39:16.759860 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 02:39:16.760004 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"mutex_wait_us":25,"spinlock_wait_cycles":5248,"thread_start_us":64,"threads_started":1}
I20260927 02:39:16.760152 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:16.754739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41159 (local address 127.0.0.1:54358)
0927 02:39:16.755009 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:16.755022 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:16.755105 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:16.755837 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:16.755843 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:16.755876 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:39:16.756092 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:16.756099 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:16.757443 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:16.757445 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:16.758607 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:16.758616 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:16.759823 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:16.759826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:16.759901 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:16.759941 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 02:39:16.760000 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 02:39:16.760050 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":71,"spinlock_wait_cycles":9088,"thread_start_us":82,"threads_started":1}
I20260927 02:39:16.760466 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:16.754501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54358 (local address 127.0.98.3:41159)
0927 02:39:16.755364 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:16.755377 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:16.755394 (+ 17us) server_negotiation.cc:407] Connection header received
0927 02:39:16.755464 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:16.755469 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:16.755668 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:16.755817 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:39:16.756204 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:16.757360 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:16.758721 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:16.759533 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:16.759631 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:16.759767 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:16.760212 (+ 445us) server_negotiation.cc:299] Negotiation successful
0927 02:39:16.760330 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":778,"thread_start_us":93,"threads_started":1}
I20260927 02:39:16.761168 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:16.754499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44417 (local address 127.0.0.1:49626)
0927 02:39:16.754872 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:16.754921 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:16.755047 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:16.755837 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:16.755843 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:16.755876 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:39:16.756092 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:16.756099 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:16.758758 (+ 2659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:16.758761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:16.759826 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:16.759835 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:16.760829 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:16.760833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:16.760916 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:16.760937 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 02:39:16.760988 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 02:39:16.761043 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":46,"spinlock_wait_cycles":13312,"thread_start_us":78,"threads_started":1}
I20260927 02:39:16.761240 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:16.754809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49626 (local address 127.0.98.1:44417)
0927 02:39:16.755354 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:16.755369 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:16.755393 (+ 24us) server_negotiation.cc:407] Connection header received
0927 02:39:16.755464 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:16.755469 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:16.755668 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:16.755817 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:39:16.757497 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:16.758675 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:16.759962 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:16.760717 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:16.760819 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:16.760965 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:16.761022 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 02:39:16.761101 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":64,"threads_started":1}
I20260927 02:39:16.763072 8327 tablet_service.cc:1467] Processing CreateTablet for tablet 75cb6f0de5ab4ca79a20a73b82b271e9 (DEFAULT_TABLE table=test-table [id=4cfa0b6441854b66ae72e855c4f16f07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:39:16.763984 8327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75cb6f0de5ab4ca79a20a73b82b271e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:39:16.763964 8187 tablet_service.cc:1467] Processing CreateTablet for tablet 75cb6f0de5ab4ca79a20a73b82b271e9 (DEFAULT_TABLE table=test-table [id=4cfa0b6441854b66ae72e855c4f16f07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:39:16.763963 8047 tablet_service.cc:1467] Processing CreateTablet for tablet 75cb6f0de5ab4ca79a20a73b82b271e9 (DEFAULT_TABLE table=test-table [id=4cfa0b6441854b66ae72e855c4f16f07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:39:16.764922 8187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75cb6f0de5ab4ca79a20a73b82b271e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:39:16.764931 8047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75cb6f0de5ab4ca79a20a73b82b271e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:39:16.770623 8421 tablet_bootstrap.cc:492] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 97e535c7163e495a87620630469ab388: Bootstrap starting.
I20260927 02:39:16.771975 8422 tablet_bootstrap.cc:492] T 75cb6f0de5ab4ca79a20a73b82b271e9 P d584780e230c4de9b74928bec112ad9a: Bootstrap starting.
I20260927 02:39:16.772560 8421 tablet_bootstrap.cc:654] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 97e535c7163e495a87620630469ab388: Neither blocks nor log segments found. Creating new log.
I20260927 02:39:16.772809 8423 tablet_bootstrap.cc:492] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 6feca25dab664c3ebcd5e224abf3e8a7: Bootstrap starting.
I20260927 02:39:16.773409 8421 log.cc:824] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 97e535c7163e495a87620630469ab388: Log is configured to *not* fsync() on all Append() calls
I20260927 02:39:16.774328 8422 tablet_bootstrap.cc:654] T 75cb6f0de5ab4ca79a20a73b82b271e9 P d584780e230c4de9b74928bec112ad9a: Neither blocks nor log segments found. Creating new log.
I20260927 02:39:16.774660 8423 tablet_bootstrap.cc:654] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 6feca25dab664c3ebcd5e224abf3e8a7: Neither blocks nor log segments found. Creating new log.
I20260927 02:39:16.775048 8422 log.cc:824] T 75cb6f0de5ab4ca79a20a73b82b271e9 P d584780e230c4de9b74928bec112ad9a: Log is configured to *not* fsync() on all Append() calls
I20260927 02:39:16.775321 8423 log.cc:824] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 6feca25dab664c3ebcd5e224abf3e8a7: Log is configured to *not* fsync() on all Append() calls
I20260927 02:39:16.775590 8421 tablet_bootstrap.cc:492] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 97e535c7163e495a87620630469ab388: No bootstrap required, opened a new log
I20260927 02:39:16.775810 8421 ts_tablet_manager.cc:1397] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 97e535c7163e495a87620630469ab388: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260927 02:39:16.776772 8422 tablet_bootstrap.cc:492] T 75cb6f0de5ab4ca79a20a73b82b271e9 P d584780e230c4de9b74928bec112ad9a: No bootstrap required, opened a new log
I20260927 02:39:16.776890 8423 tablet_bootstrap.cc:492] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 6feca25dab664c3ebcd5e224abf3e8a7: No bootstrap required, opened a new log
I20260927 02:39:16.776944 8422 ts_tablet_manager.cc:1397] T 75cb6f0de5ab4ca79a20a73b82b271e9 P d584780e230c4de9b74928bec112ad9a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260927 02:39:16.777083 8423 ts_tablet_manager.cc:1397] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 6feca25dab664c3ebcd5e224abf3e8a7: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260927 02:39:16.779508 8421 raft_consensus.cc:348] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 97e535c7163e495a87620630469ab388 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } } peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } }
I20260927 02:39:16.779754 8421 raft_consensus.cc:374] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 97e535c7163e495a87620630469ab388 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:39:16.779809 8421 raft_consensus.cc:729] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 97e535c7163e495a87620630469ab388 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97e535c7163e495a87620630469ab388, State: Initialized, Role: FOLLOWER
I20260927 02:39:16.779736 8423 raft_consensus.cc:348] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 6feca25dab664c3ebcd5e224abf3e8a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } } peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } }
I20260927 02:39:16.779933 8423 raft_consensus.cc:374] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 6feca25dab664c3ebcd5e224abf3e8a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:39:16.779963 8423 raft_consensus.cc:729] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 6feca25dab664c3ebcd5e224abf3e8a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6feca25dab664c3ebcd5e224abf3e8a7, State: Initialized, Role: FOLLOWER
I20260927 02:39:16.780221 8421 consensus_queue.cc:260] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 97e535c7163e495a87620630469ab388 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } } peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } }
I20260927 02:39:16.780364 8423 consensus_queue.cc:260] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 6feca25dab664c3ebcd5e224abf3e8a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } } peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } }
I20260927 02:39:16.780380 8422 raft_consensus.cc:348] T 75cb6f0de5ab4ca79a20a73b82b271e9 P d584780e230c4de9b74928bec112ad9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } } peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } }
I20260927 02:39:16.780613 8422 raft_consensus.cc:374] T 75cb6f0de5ab4ca79a20a73b82b271e9 P d584780e230c4de9b74928bec112ad9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:39:16.780700 8422 raft_consensus.cc:729] T 75cb6f0de5ab4ca79a20a73b82b271e9 P d584780e230c4de9b74928bec112ad9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d584780e230c4de9b74928bec112ad9a, State: Initialized, Role: FOLLOWER
I20260927 02:39:16.780937 8253 heartbeater.cc:503] Master 127.0.98.62:37181 was elected leader, sending a full tablet report...
I20260927 02:39:16.781172 8421 ts_tablet_manager.cc:1428] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 97e535c7163e495a87620630469ab388: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 02:39:16.781190 8423 ts_tablet_manager.cc:1428] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 6feca25dab664c3ebcd5e224abf3e8a7: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260927 02:39:16.781481 8393 heartbeater.cc:503] Master 127.0.98.62:37181 was elected leader, sending a full tablet report...
I20260927 02:39:16.781788 8422 consensus_queue.cc:260] T 75cb6f0de5ab4ca79a20a73b82b271e9 P d584780e230c4de9b74928bec112ad9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } } peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } }
I20260927 02:39:16.782428 8113 heartbeater.cc:503] Master 127.0.98.62:37181 was elected leader, sending a full tablet report...
I20260927 02:39:16.782620 8422 ts_tablet_manager.cc:1428] T 75cb6f0de5ab4ca79a20a73b82b271e9 P d584780e230c4de9b74928bec112ad9a: Time spent starting tablet: real 0.006s user 0.000s sys 0.006s
I20260927 02:39:16.874725 8428 raft_consensus.cc:482] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 6feca25dab664c3ebcd5e224abf3e8a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:39:16.874922 8428 raft_consensus.cc:504] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 6feca25dab664c3ebcd5e224abf3e8a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } } peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } }
I20260927 02:39:16.876047 8428 leader_election.cc:288] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 6feca25dab664c3ebcd5e224abf3e8a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97e535c7163e495a87620630469ab388 (127.0.98.3:41159), d584780e230c4de9b74928bec112ad9a (127.0.98.1:44417)
I20260927 02:39:16.879681 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:16.876371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58093 (local address 127.0.98.1:44417)
0927 02:39:16.876487 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:16.876491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:16.876640 (+ 149us) server_negotiation.cc:407] Connection header received
0927 02:39:16.876799 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:16.876801 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:16.876843 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:16.876929 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:39:16.877184 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:16.877707 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:16.878768 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:16.879168 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:16.879257 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:16.879318 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:16.879503 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 02:39:16.879569 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 02:39:16.879693 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:16.876275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44417 (local address 127.0.98.2:58093)
0927 02:39:16.876567 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:16.876579 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:16.876701 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:16.876925 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:16.876928 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:16.876942 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:39:16.877091 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:16.877097 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:16.877818 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:16.877821 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:16.878650 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:16.878659 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:16.879275 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:16.879278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:16.879386 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:16.879410 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 02:39:16.879475 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 02:39:16.879546 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":20,"spinlock_wait_cycles":6400,"thread_start_us":106,"threads_started":1}
I20260927 02:39:16.880026 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:16.876358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46463 (local address 127.0.98.3:41159)
0927 02:39:16.876488 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:16.876493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:16.876647 (+ 154us) server_negotiation.cc:407] Connection header received
0927 02:39:16.876820 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:16.876822 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:16.876865 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:16.876928 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:39:16.877215 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:16.877706 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:16.879117 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:16.879501 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:16.879591 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:16.879647 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:16.879840 (+ 193us) server_negotiation.cc:299] Negotiation successful
0927 02:39:16.879915 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 02:39:16.880295 8067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75cb6f0de5ab4ca79a20a73b82b271e9" candidate_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d584780e230c4de9b74928bec112ad9a" is_pre_election: true
I20260927 02:39:16.880376 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:16.876255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41159 (local address 127.0.98.2:46463)
0927 02:39:16.876554 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:16.876567 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:16.876698 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:16.876952 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:16.876955 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:16.876970 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:39:16.877114 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:16.877119 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:16.877815 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:16.877818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:16.878996 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:16.879006 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:16.879608 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:16.879611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:16.879703 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:16.879721 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 02:39:16.880207 (+ 486us) client_negotiation.cc:240] Negotiation successful
0927 02:39:16.880279 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":21,"spinlock_wait_cycles":42112}
I20260927 02:39:16.880626 8067 raft_consensus.cc:2454] T 75cb6f0de5ab4ca79a20a73b82b271e9 P d584780e230c4de9b74928bec112ad9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6feca25dab664c3ebcd5e224abf3e8a7 in term 0.
I20260927 02:39:16.880955 8347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75cb6f0de5ab4ca79a20a73b82b271e9" candidate_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97e535c7163e495a87620630469ab388" is_pre_election: true
I20260927 02:39:16.881099 8139 leader_election.cc:302] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 6feca25dab664c3ebcd5e224abf3e8a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6feca25dab664c3ebcd5e224abf3e8a7, d584780e230c4de9b74928bec112ad9a; no voters:
I20260927 02:39:16.881220 8347 raft_consensus.cc:2454] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 97e535c7163e495a87620630469ab388 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6feca25dab664c3ebcd5e224abf3e8a7 in term 0.
I20260927 02:39:16.881371 8428 raft_consensus.cc:2781] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 6feca25dab664c3ebcd5e224abf3e8a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:39:16.881456 8428 raft_consensus.cc:482] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 6feca25dab664c3ebcd5e224abf3e8a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:39:16.881510 8428 raft_consensus.cc:3037] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 6feca25dab664c3ebcd5e224abf3e8a7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:39:16.882546 8428 raft_consensus.cc:504] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 6feca25dab664c3ebcd5e224abf3e8a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } } peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } }
I20260927 02:39:16.882824 8428 leader_election.cc:288] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 6feca25dab664c3ebcd5e224abf3e8a7 [CANDIDATE]: Term 1 election: Requested vote from peers 97e535c7163e495a87620630469ab388 (127.0.98.3:41159), d584780e230c4de9b74928bec112ad9a (127.0.98.1:44417)
I20260927 02:39:16.883181 8347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75cb6f0de5ab4ca79a20a73b82b271e9" candidate_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97e535c7163e495a87620630469ab388"
I20260927 02:39:16.883306 8067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75cb6f0de5ab4ca79a20a73b82b271e9" candidate_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d584780e230c4de9b74928bec112ad9a"
I20260927 02:39:16.883303 8347 raft_consensus.cc:3037] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 97e535c7163e495a87620630469ab388 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:39:16.883464 8067 raft_consensus.cc:3037] T 75cb6f0de5ab4ca79a20a73b82b271e9 P d584780e230c4de9b74928bec112ad9a [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:39:16.884541 8067 raft_consensus.cc:2454] T 75cb6f0de5ab4ca79a20a73b82b271e9 P d584780e230c4de9b74928bec112ad9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6feca25dab664c3ebcd5e224abf3e8a7 in term 1.
I20260927 02:39:16.884541 8347 raft_consensus.cc:2454] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 97e535c7163e495a87620630469ab388 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6feca25dab664c3ebcd5e224abf3e8a7 in term 1.
I20260927 02:39:16.884819 8138 leader_election.cc:302] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 6feca25dab664c3ebcd5e224abf3e8a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6feca25dab664c3ebcd5e224abf3e8a7, 97e535c7163e495a87620630469ab388; no voters:
I20260927 02:39:16.884977 8428 raft_consensus.cc:2781] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 6feca25dab664c3ebcd5e224abf3e8a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:39:16.885164 8428 raft_consensus.cc:686] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 6feca25dab664c3ebcd5e224abf3e8a7 [term 1 LEADER]: Becoming Leader. State: Replica: 6feca25dab664c3ebcd5e224abf3e8a7, State: Running, Role: LEADER
I20260927 02:39:16.885456 8428 consensus_queue.cc:237] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 6feca25dab664c3ebcd5e224abf3e8a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } } peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } }
I20260927 02:39:16.888389 7910 catalog_manager.cc:5495] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 6feca25dab664c3ebcd5e224abf3e8a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6feca25dab664c3ebcd5e224abf3e8a7 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } health_report { overall_health: UNKNOWN } } }
I20260927 02:39:16.910303 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:16.907227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45606 (local address 127.0.98.2:39041)
0927 02:39:16.907369 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:16.907372 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:16.907387 (+ 15us) server_negotiation.cc:407] Connection header received
0927 02:39:16.907456 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:16.907458 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:16.907511 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:16.907597 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:39:16.907772 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:16.908258 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:16.909046 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:16.909261 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:16.909389 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:16.909546 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:39:16.909964 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:39:16.910044 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:16.910105 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 02:39:16.910154 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 02:39:16.915560 8347 raft_consensus.cc:1261] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 97e535c7163e495a87620630469ab388 [term 1 FOLLOWER]: Refusing update from remote peer 6feca25dab664c3ebcd5e224abf3e8a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:39:16.915560 8067 raft_consensus.cc:1261] T 75cb6f0de5ab4ca79a20a73b82b271e9 P d584780e230c4de9b74928bec112ad9a [term 1 FOLLOWER]: Refusing update from remote peer 6feca25dab664c3ebcd5e224abf3e8a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:39:16.916145 8431 consensus_queue.cc:1035] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 6feca25dab664c3ebcd5e224abf3e8a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:39:16.916311 8428 consensus_queue.cc:1035] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 6feca25dab664c3ebcd5e224abf3e8a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:39:16.923101 8434 mvcc.cc:205] Tried to move back new op lower bound from 7333792796319674368 to 7333792796204937216. Current Snapshot: MvccSnapshot[applied={T|T < 7333792796319674368}]
I20260927 02:39:16.923693 8436 mvcc.cc:205] Tried to move back new op lower bound from 7333792796319674368 to 7333792796204937216. Current Snapshot: MvccSnapshot[applied={T|T < 7333792796319674368}]
I20260927 02:39:16.925992 8435 mvcc.cc:205] Tried to move back new op lower bound from 7333792796319674368 to 7333792796204937216. Current Snapshot: MvccSnapshot[applied={T|T < 7333792796319674368}]
I20260927 02:39:16.930546 7908 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:39:16.930688 7908 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:39:16.932780 7908 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c03c951086c4af5b892870108fd8990: Sending DeleteTablet for 3 replicas of tablet 75cb6f0de5ab4ca79a20a73b82b271e9
I20260927 02:39:16.933475 8187 tablet_service.cc:1514] Processing DeleteTablet for tablet 75cb6f0de5ab4ca79a20a73b82b271e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:39:16 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45594
I20260927 02:39:16.933609 8047 tablet_service.cc:1514] Processing DeleteTablet for tablet 75cb6f0de5ab4ca79a20a73b82b271e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:39:16 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49626
I20260927 02:39:16.933602 8327 tablet_service.cc:1514] Processing DeleteTablet for tablet 75cb6f0de5ab4ca79a20a73b82b271e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:39:16 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54358
I20260927 02:39:16.934003 8444 tablet_replica.cc:330] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 6feca25dab664c3ebcd5e224abf3e8a7: stopping tablet replica
I20260927 02:39:16.934142 8446 tablet_replica.cc:330] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 97e535c7163e495a87620630469ab388: stopping tablet replica
I20260927 02:39:16.934396 8444 raft_consensus.cc:2229] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 6feca25dab664c3ebcd5e224abf3e8a7 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:39:16.934497 8446 raft_consensus.cc:2229] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 97e535c7163e495a87620630469ab388 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:39:16.934787 8444 raft_consensus.cc:2258] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 6feca25dab664c3ebcd5e224abf3e8a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:39:16.934782 7908 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35698:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:39:16.935154 7908 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:39:16.936043 8446 raft_consensus.cc:2258] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 97e535c7163e495a87620630469ab388 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:39:16.936195 8444 ts_tablet_manager.cc:1905] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 6feca25dab664c3ebcd5e224abf3e8a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:39:16.936635 8445 tablet_replica.cc:330] T 75cb6f0de5ab4ca79a20a73b82b271e9 P d584780e230c4de9b74928bec112ad9a: stopping tablet replica
I20260927 02:39:16.939294 8444 ts_tablet_manager.cc:1918] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 6feca25dab664c3ebcd5e224abf3e8a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:39:16.939464 8444 log.cc:1196] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 6feca25dab664c3ebcd5e224abf3e8a7: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-1/wal/wals/75cb6f0de5ab4ca79a20a73b82b271e9
I20260927 02:39:16.939538 8187 tablet_service.cc:1467] Processing CreateTablet for tablet ac045a4e76dc463e82b3c1c19458fb8a (DEFAULT_TABLE table=test-table [id=7c0e7bf0b98947e08d1402fc33967ece]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:39:16.939749 8444 ts_tablet_manager.cc:1939] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 6feca25dab664c3ebcd5e224abf3e8a7: Deleting consensus metadata
I20260927 02:39:16.939764 8187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac045a4e76dc463e82b3c1c19458fb8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:39:16.940297 8327 tablet_service.cc:1467] Processing CreateTablet for tablet ac045a4e76dc463e82b3c1c19458fb8a (DEFAULT_TABLE table=test-table [id=7c0e7bf0b98947e08d1402fc33967ece]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:39:16.940519 8327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac045a4e76dc463e82b3c1c19458fb8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:39:16.941798 8423 tablet_bootstrap.cc:492] T ac045a4e76dc463e82b3c1c19458fb8a P 6feca25dab664c3ebcd5e224abf3e8a7: Bootstrap starting.
I20260927 02:39:16.942788 8423 tablet_bootstrap.cc:654] T ac045a4e76dc463e82b3c1c19458fb8a P 6feca25dab664c3ebcd5e224abf3e8a7: Neither blocks nor log segments found. Creating new log.
I20260927 02:39:16.942793 8421 tablet_bootstrap.cc:492] T ac045a4e76dc463e82b3c1c19458fb8a P 97e535c7163e495a87620630469ab388: Bootstrap starting.
I20260927 02:39:16.943671 8421 tablet_bootstrap.cc:654] T ac045a4e76dc463e82b3c1c19458fb8a P 97e535c7163e495a87620630469ab388: Neither blocks nor log segments found. Creating new log.
I20260927 02:39:16.943934 8423 tablet_bootstrap.cc:492] T ac045a4e76dc463e82b3c1c19458fb8a P 6feca25dab664c3ebcd5e224abf3e8a7: No bootstrap required, opened a new log
I20260927 02:39:16.944003 8423 ts_tablet_manager.cc:1397] T ac045a4e76dc463e82b3c1c19458fb8a P 6feca25dab664c3ebcd5e224abf3e8a7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:39:16.944330 7897 catalog_manager.cc:4841] TS 6feca25dab664c3ebcd5e224abf3e8a7 (127.0.98.2:39041): tablet 75cb6f0de5ab4ca79a20a73b82b271e9 (table test-table [id=4cfa0b6441854b66ae72e855c4f16f07]) successfully deleted
I20260927 02:39:16.944415 8423 raft_consensus.cc:348] T ac045a4e76dc463e82b3c1c19458fb8a P 6feca25dab664c3ebcd5e224abf3e8a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } } peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } }
I20260927 02:39:16.944511 8423 raft_consensus.cc:374] T ac045a4e76dc463e82b3c1c19458fb8a P 6feca25dab664c3ebcd5e224abf3e8a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:39:16.944545 8423 raft_consensus.cc:729] T ac045a4e76dc463e82b3c1c19458fb8a P 6feca25dab664c3ebcd5e224abf3e8a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6feca25dab664c3ebcd5e224abf3e8a7, State: Initialized, Role: FOLLOWER
I20260927 02:39:16.944649 8423 consensus_queue.cc:260] T ac045a4e76dc463e82b3c1c19458fb8a P 6feca25dab664c3ebcd5e224abf3e8a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } } peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } }
I20260927 02:39:16.944753 8421 tablet_bootstrap.cc:492] T ac045a4e76dc463e82b3c1c19458fb8a P 97e535c7163e495a87620630469ab388: No bootstrap required, opened a new log
I20260927 02:39:16.944823 8421 ts_tablet_manager.cc:1397] T ac045a4e76dc463e82b3c1c19458fb8a P 97e535c7163e495a87620630469ab388: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:39:16.944895 8423 ts_tablet_manager.cc:1428] T ac045a4e76dc463e82b3c1c19458fb8a P 6feca25dab664c3ebcd5e224abf3e8a7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:39:16.945214 8421 raft_consensus.cc:348] T ac045a4e76dc463e82b3c1c19458fb8a P 97e535c7163e495a87620630469ab388 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } } peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } }
I20260927 02:39:16.945313 8421 raft_consensus.cc:374] T ac045a4e76dc463e82b3c1c19458fb8a P 97e535c7163e495a87620630469ab388 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:39:16.945341 8421 raft_consensus.cc:729] T ac045a4e76dc463e82b3c1c19458fb8a P 97e535c7163e495a87620630469ab388 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97e535c7163e495a87620630469ab388, State: Initialized, Role: FOLLOWER
I20260927 02:39:16.945441 8421 consensus_queue.cc:260] T ac045a4e76dc463e82b3c1c19458fb8a P 97e535c7163e495a87620630469ab388 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } } peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } }
I20260927 02:39:16.945667 8421 ts_tablet_manager.cc:1428] T ac045a4e76dc463e82b3c1c19458fb8a P 97e535c7163e495a87620630469ab388: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:39:16.946501 8446 ts_tablet_manager.cc:1905] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 97e535c7163e495a87620630469ab388: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 02:39:16.947275 8394 tablet.cc:2369] T ac045a4e76dc463e82b3c1c19458fb8a P 97e535c7163e495a87620630469ab388: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:39:16.949012 8446 ts_tablet_manager.cc:1918] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 97e535c7163e495a87620630469ab388: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:39:16.949087 8446 log.cc:1196] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 97e535c7163e495a87620630469ab388: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-2/wal/wals/75cb6f0de5ab4ca79a20a73b82b271e9
I20260927 02:39:16.949306 8446 ts_tablet_manager.cc:1939] T 75cb6f0de5ab4ca79a20a73b82b271e9 P 97e535c7163e495a87620630469ab388: Deleting consensus metadata
I20260927 02:39:16.939396 8047 tablet_service.cc:1467] Processing CreateTablet for tablet ac045a4e76dc463e82b3c1c19458fb8a (DEFAULT_TABLE table=test-table [id=7c0e7bf0b98947e08d1402fc33967ece]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:39:16.949899 8047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac045a4e76dc463e82b3c1c19458fb8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:39:16.950603 8445 raft_consensus.cc:2229] T 75cb6f0de5ab4ca79a20a73b82b271e9 P d584780e230c4de9b74928bec112ad9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:39:16.950925 8445 raft_consensus.cc:2258] T 75cb6f0de5ab4ca79a20a73b82b271e9 P d584780e230c4de9b74928bec112ad9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:39:16.952196 8422 tablet_bootstrap.cc:492] T ac045a4e76dc463e82b3c1c19458fb8a P d584780e230c4de9b74928bec112ad9a: Bootstrap starting.
I20260927 02:39:16.952332 8445 ts_tablet_manager.cc:1905] T 75cb6f0de5ab4ca79a20a73b82b271e9 P d584780e230c4de9b74928bec112ad9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:39:16.952481 7894 catalog_manager.cc:4841] TS 97e535c7163e495a87620630469ab388 (127.0.98.3:41159): tablet 75cb6f0de5ab4ca79a20a73b82b271e9 (table test-table [id=4cfa0b6441854b66ae72e855c4f16f07]) successfully deleted
I20260927 02:39:16.953130 8422 tablet_bootstrap.cc:654] T ac045a4e76dc463e82b3c1c19458fb8a P d584780e230c4de9b74928bec112ad9a: Neither blocks nor log segments found. Creating new log.
I20260927 02:39:16.954149 8422 tablet_bootstrap.cc:492] T ac045a4e76dc463e82b3c1c19458fb8a P d584780e230c4de9b74928bec112ad9a: No bootstrap required, opened a new log
I20260927 02:39:16.954222 8422 ts_tablet_manager.cc:1397] T ac045a4e76dc463e82b3c1c19458fb8a P d584780e230c4de9b74928bec112ad9a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:39:16.954622 8445 ts_tablet_manager.cc:1918] T 75cb6f0de5ab4ca79a20a73b82b271e9 P d584780e230c4de9b74928bec112ad9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:39:16.954708 8445 log.cc:1196] T 75cb6f0de5ab4ca79a20a73b82b271e9 P d584780e230c4de9b74928bec112ad9a: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-0/wal/wals/75cb6f0de5ab4ca79a20a73b82b271e9
I20260927 02:39:16.954721 8422 raft_consensus.cc:348] T ac045a4e76dc463e82b3c1c19458fb8a P d584780e230c4de9b74928bec112ad9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } } peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } }
I20260927 02:39:16.954830 8422 raft_consensus.cc:374] T ac045a4e76dc463e82b3c1c19458fb8a P d584780e230c4de9b74928bec112ad9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:39:16.954876 8422 raft_consensus.cc:729] T ac045a4e76dc463e82b3c1c19458fb8a P d584780e230c4de9b74928bec112ad9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d584780e230c4de9b74928bec112ad9a, State: Initialized, Role: FOLLOWER
I20260927 02:39:16.955013 8445 ts_tablet_manager.cc:1939] T 75cb6f0de5ab4ca79a20a73b82b271e9 P d584780e230c4de9b74928bec112ad9a: Deleting consensus metadata
I20260927 02:39:16.955013 8422 consensus_queue.cc:260] T ac045a4e76dc463e82b3c1c19458fb8a P d584780e230c4de9b74928bec112ad9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } } peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } }
I20260927 02:39:16.955292 8422 ts_tablet_manager.cc:1428] T ac045a4e76dc463e82b3c1c19458fb8a P d584780e230c4de9b74928bec112ad9a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:39:16.956125 7895 catalog_manager.cc:4841] TS d584780e230c4de9b74928bec112ad9a (127.0.98.1:44417): tablet 75cb6f0de5ab4ca79a20a73b82b271e9 (table test-table [id=4cfa0b6441854b66ae72e855c4f16f07]) successfully deleted
W20260927 02:39:16.971161 8254 tablet.cc:2369] T ac045a4e76dc463e82b3c1c19458fb8a P 6feca25dab664c3ebcd5e224abf3e8a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:39:17.011803 8114 tablet.cc:2369] T ac045a4e76dc463e82b3c1c19458fb8a P d584780e230c4de9b74928bec112ad9a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:39:17.335582 8427 raft_consensus.cc:482] T ac045a4e76dc463e82b3c1c19458fb8a P 97e535c7163e495a87620630469ab388 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:39:17.335739 8427 raft_consensus.cc:504] T ac045a4e76dc463e82b3c1c19458fb8a P 97e535c7163e495a87620630469ab388 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } } peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } }
I20260927 02:39:17.336809 8427 leader_election.cc:288] T ac045a4e76dc463e82b3c1c19458fb8a P 97e535c7163e495a87620630469ab388 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d584780e230c4de9b74928bec112ad9a (127.0.98.1:44417), 6feca25dab664c3ebcd5e224abf3e8a7 (127.0.98.2:39041)
I20260927 02:39:17.341912 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:17.337140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38663 (local address 127.0.98.2:39041)
0927 02:39:17.337272 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:17.337277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:17.337491 (+ 214us) server_negotiation.cc:407] Connection header received
0927 02:39:17.337646 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:17.337649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:17.337703 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:17.337784 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:39:17.338151 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:17.338977 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:17.340717 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:17.341355 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:17.341454 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:17.341553 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:17.341710 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 02:39:17.341779 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 02:39:17.341912 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:17.337164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44835 (local address 127.0.98.1:44417)
0927 02:39:17.337393 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:17.337396 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:17.337526 (+ 130us) server_negotiation.cc:407] Connection header received
0927 02:39:17.337679 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:17.337681 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:17.337724 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:17.337788 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:39:17.338151 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:17.338977 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:17.340717 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:17.341341 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:17.341446 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:17.341511 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:17.341710 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 02:39:17.341779 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260927 02:39:17.341979 8450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:17.337074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39041 (local address 127.0.98.3:38663)
0927 02:39:17.337384 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:17.337399 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:17.337546 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:17.337828 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:17.337832 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:17.337863 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:39:17.338033 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:17.338041 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:17.339113 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:17.339137 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:17.340581 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:17.340592 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:17.341469 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:17.341473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:17.341575 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:17.341607 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 02:39:17.341685 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 02:39:17.341779 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":133,"spinlock_wait_cycles":42752,"thread_start_us":65,"threads_started":1}
I20260927 02:39:17.341979 8451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:17.337074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44417 (local address 127.0.98.3:44835)
0927 02:39:17.337414 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:17.337428 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:17.337546 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:17.337828 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:17.337832 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:17.337860 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:39:17.338042 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:17.338048 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:17.339113 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:17.339117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:17.340581 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:17.340592 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:17.341482 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:17.341485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:17.341608 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:17.341625 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 02:39:17.341685 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 02:39:17.341784 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":172,"spinlock_wait_cycles":17408,"thread_start_us":88,"threads_started":1}
I20260927 02:39:17.342702 8067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac045a4e76dc463e82b3c1c19458fb8a" candidate_uuid: "97e535c7163e495a87620630469ab388" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d584780e230c4de9b74928bec112ad9a" is_pre_election: true
I20260927 02:39:17.342800 8207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac045a4e76dc463e82b3c1c19458fb8a" candidate_uuid: "97e535c7163e495a87620630469ab388" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" is_pre_election: true
I20260927 02:39:17.342857 8067 raft_consensus.cc:2454] T ac045a4e76dc463e82b3c1c19458fb8a P d584780e230c4de9b74928bec112ad9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97e535c7163e495a87620630469ab388 in term 0.
I20260927 02:39:17.343043 8207 raft_consensus.cc:2454] T ac045a4e76dc463e82b3c1c19458fb8a P 6feca25dab664c3ebcd5e224abf3e8a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97e535c7163e495a87620630469ab388 in term 0.
I20260927 02:39:17.343186 8279 leader_election.cc:302] T ac045a4e76dc463e82b3c1c19458fb8a P 97e535c7163e495a87620630469ab388 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97e535c7163e495a87620630469ab388, d584780e230c4de9b74928bec112ad9a; no voters:
I20260927 02:39:17.343482 8427 raft_consensus.cc:2781] T ac045a4e76dc463e82b3c1c19458fb8a P 97e535c7163e495a87620630469ab388 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:39:17.343550 8427 raft_consensus.cc:482] T ac045a4e76dc463e82b3c1c19458fb8a P 97e535c7163e495a87620630469ab388 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:39:17.343576 8427 raft_consensus.cc:3037] T ac045a4e76dc463e82b3c1c19458fb8a P 97e535c7163e495a87620630469ab388 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:39:17.344492 8427 raft_consensus.cc:504] T ac045a4e76dc463e82b3c1c19458fb8a P 97e535c7163e495a87620630469ab388 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } } peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } }
I20260927 02:39:17.344813 8427 leader_election.cc:288] T ac045a4e76dc463e82b3c1c19458fb8a P 97e535c7163e495a87620630469ab388 [CANDIDATE]: Term 1 election: Requested vote from peers d584780e230c4de9b74928bec112ad9a (127.0.98.1:44417), 6feca25dab664c3ebcd5e224abf3e8a7 (127.0.98.2:39041)
I20260927 02:39:17.345166 8067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac045a4e76dc463e82b3c1c19458fb8a" candidate_uuid: "97e535c7163e495a87620630469ab388" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d584780e230c4de9b74928bec112ad9a"
I20260927 02:39:17.345166 8207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac045a4e76dc463e82b3c1c19458fb8a" candidate_uuid: "97e535c7163e495a87620630469ab388" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6feca25dab664c3ebcd5e224abf3e8a7"
I20260927 02:39:17.345268 8207 raft_consensus.cc:3037] T ac045a4e76dc463e82b3c1c19458fb8a P 6feca25dab664c3ebcd5e224abf3e8a7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:39:17.345275 8067 raft_consensus.cc:3037] T ac045a4e76dc463e82b3c1c19458fb8a P d584780e230c4de9b74928bec112ad9a [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:39:17.346163 8067 raft_consensus.cc:2454] T ac045a4e76dc463e82b3c1c19458fb8a P d584780e230c4de9b74928bec112ad9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97e535c7163e495a87620630469ab388 in term 1.
I20260927 02:39:17.346285 8207 raft_consensus.cc:2454] T ac045a4e76dc463e82b3c1c19458fb8a P 6feca25dab664c3ebcd5e224abf3e8a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97e535c7163e495a87620630469ab388 in term 1.
I20260927 02:39:17.346434 8279 leader_election.cc:302] T ac045a4e76dc463e82b3c1c19458fb8a P 97e535c7163e495a87620630469ab388 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97e535c7163e495a87620630469ab388, d584780e230c4de9b74928bec112ad9a; no voters:
I20260927 02:39:17.346576 8427 raft_consensus.cc:2781] T ac045a4e76dc463e82b3c1c19458fb8a P 97e535c7163e495a87620630469ab388 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:39:17.346735 8427 raft_consensus.cc:686] T ac045a4e76dc463e82b3c1c19458fb8a P 97e535c7163e495a87620630469ab388 [term 1 LEADER]: Becoming Leader. State: Replica: 97e535c7163e495a87620630469ab388, State: Running, Role: LEADER
I20260927 02:39:17.346972 8427 consensus_queue.cc:237] T ac045a4e76dc463e82b3c1c19458fb8a P 97e535c7163e495a87620630469ab388 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } } peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } }
I20260927 02:39:17.349040 7910 catalog_manager.cc:5495] T ac045a4e76dc463e82b3c1c19458fb8a P 97e535c7163e495a87620630469ab388 reported cstate change: term changed from 0 to 1, leader changed from <none> to 97e535c7163e495a87620630469ab388 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "97e535c7163e495a87620630469ab388" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } health_report { overall_health: HEALTHY } } }
Sep 27 02:39:17 dist-test-slave-zv8z krb5kdc[7865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476755, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:39:17.423846 8459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:17.415942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37181 (local address 127.0.0.1:35700)
0927 02:39:17.416211 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:17.416224 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:17.416328 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:17.416602 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:17.416604 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:17.417002 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:17.417093 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:17.417099 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:17.417850 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:17.417853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:17.418494 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:17.418505 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:17.418918 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:17.418922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:17.419003 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:17.419531 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:17.419543 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:17.420932 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:17.422413 (+ 1481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:17.422420 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:17.422425 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:17.422656 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:17.422898 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:17.422900 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:17.422901 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:17.422988 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:17.423287 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:17.423294 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:17.423418 (+ 124us) client_negotiation.cc:769] Sending connection context
0927 02:39:17.423528 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 02:39:17.423664 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":90,"threads_started":1}
I20260927 02:39:17.423866 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:17.415977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35700 (local address 127.0.98.62:37181)
0927 02:39:17.416230 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:17.416236 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:17.416279 (+ 43us) server_negotiation.cc:407] Connection header received
0927 02:39:17.416453 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:17.416455 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:17.416497 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:17.416574 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:17.417212 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:17.417769 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:17.418597 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:17.418819 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:17.418899 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:17.421077 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:17.421080 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:17.421082 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:17.421101 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:17.422317 (+ 1216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:17.422738 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:17.422741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:17.422742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:17.422825 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:17.423051 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:17.423054 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:17.423055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:17.423205 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:17.423273 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:17.423602 (+ 329us) server_negotiation.cc:299] Negotiation successful
0927 02:39:17.423719 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":188,"thread_start_us":104,"threads_started":1}
I20260927 02:39:17.427690 7910 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35700:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 02:39:17.428402 7910 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:39:17.432914 8047 tablet_service.cc:1467] Processing CreateTablet for tablet 524afaa00d134834b85385c9dfa2534d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=85fcecf2ebe544f49ce71a77b6c373c4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:39:17.432963 8187 tablet_service.cc:1467] Processing CreateTablet for tablet 524afaa00d134834b85385c9dfa2534d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=85fcecf2ebe544f49ce71a77b6c373c4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:39:17.433301 8047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 524afaa00d134834b85385c9dfa2534d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:39:17.433197 8327 tablet_service.cc:1467] Processing CreateTablet for tablet 524afaa00d134834b85385c9dfa2534d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=85fcecf2ebe544f49ce71a77b6c373c4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:39:17.433339 8187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 524afaa00d134834b85385c9dfa2534d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:39:17.433554 8327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 524afaa00d134834b85385c9dfa2534d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:39:17.435672 8423 tablet_bootstrap.cc:492] T 524afaa00d134834b85385c9dfa2534d P 6feca25dab664c3ebcd5e224abf3e8a7: Bootstrap starting.
I20260927 02:39:17.436110 8422 tablet_bootstrap.cc:492] T 524afaa00d134834b85385c9dfa2534d P d584780e230c4de9b74928bec112ad9a: Bootstrap starting.
I20260927 02:39:17.436460 8423 tablet_bootstrap.cc:654] T 524afaa00d134834b85385c9dfa2534d P 6feca25dab664c3ebcd5e224abf3e8a7: Neither blocks nor log segments found. Creating new log.
I20260927 02:39:17.436940 8422 tablet_bootstrap.cc:654] T 524afaa00d134834b85385c9dfa2534d P d584780e230c4de9b74928bec112ad9a: Neither blocks nor log segments found. Creating new log.
I20260927 02:39:17.436996 8421 tablet_bootstrap.cc:492] T 524afaa00d134834b85385c9dfa2534d P 97e535c7163e495a87620630469ab388: Bootstrap starting.
I20260927 02:39:17.437614 8423 tablet_bootstrap.cc:492] T 524afaa00d134834b85385c9dfa2534d P 6feca25dab664c3ebcd5e224abf3e8a7: No bootstrap required, opened a new log
I20260927 02:39:17.437705 8423 ts_tablet_manager.cc:1397] T 524afaa00d134834b85385c9dfa2534d P 6feca25dab664c3ebcd5e224abf3e8a7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:39:17.437862 8421 tablet_bootstrap.cc:654] T 524afaa00d134834b85385c9dfa2534d P 97e535c7163e495a87620630469ab388: Neither blocks nor log segments found. Creating new log.
I20260927 02:39:17.438125 8423 raft_consensus.cc:348] T 524afaa00d134834b85385c9dfa2534d P 6feca25dab664c3ebcd5e224abf3e8a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } } peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } }
I20260927 02:39:17.438212 8423 raft_consensus.cc:374] T 524afaa00d134834b85385c9dfa2534d P 6feca25dab664c3ebcd5e224abf3e8a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:39:17.438231 8423 raft_consensus.cc:729] T 524afaa00d134834b85385c9dfa2534d P 6feca25dab664c3ebcd5e224abf3e8a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6feca25dab664c3ebcd5e224abf3e8a7, State: Initialized, Role: FOLLOWER
I20260927 02:39:17.438330 8422 tablet_bootstrap.cc:492] T 524afaa00d134834b85385c9dfa2534d P d584780e230c4de9b74928bec112ad9a: No bootstrap required, opened a new log
I20260927 02:39:17.438350 8423 consensus_queue.cc:260] T 524afaa00d134834b85385c9dfa2534d P 6feca25dab664c3ebcd5e224abf3e8a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } } peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } }
I20260927 02:39:17.438405 8422 ts_tablet_manager.cc:1397] T 524afaa00d134834b85385c9dfa2534d P d584780e230c4de9b74928bec112ad9a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:39:17.438576 8423 ts_tablet_manager.cc:1428] T 524afaa00d134834b85385c9dfa2534d P 6feca25dab664c3ebcd5e224abf3e8a7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:39:17.438558 8431 tablet_replica.cc:439] T 524afaa00d134834b85385c9dfa2534d P 6feca25dab664c3ebcd5e224abf3e8a7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } } peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } } }
I20260927 02:39:17.438674 8431 tablet_replica.cc:442] T 524afaa00d134834b85385c9dfa2534d P 6feca25dab664c3ebcd5e224abf3e8a7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:39:17.438830 8422 raft_consensus.cc:348] T 524afaa00d134834b85385c9dfa2534d P d584780e230c4de9b74928bec112ad9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } } peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } }
I20260927 02:39:17.438931 8422 raft_consensus.cc:374] T 524afaa00d134834b85385c9dfa2534d P d584780e230c4de9b74928bec112ad9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:39:17.438962 8422 raft_consensus.cc:729] T 524afaa00d134834b85385c9dfa2534d P d584780e230c4de9b74928bec112ad9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d584780e230c4de9b74928bec112ad9a, State: Initialized, Role: FOLLOWER
I20260927 02:39:17.439074 8422 consensus_queue.cc:260] T 524afaa00d134834b85385c9dfa2534d P d584780e230c4de9b74928bec112ad9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } } peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } }
I20260927 02:39:17.439303 8421 tablet_bootstrap.cc:492] T 524afaa00d134834b85385c9dfa2534d P 97e535c7163e495a87620630469ab388: No bootstrap required, opened a new log
I20260927 02:39:17.439299 8429 tablet_replica.cc:439] T 524afaa00d134834b85385c9dfa2534d P d584780e230c4de9b74928bec112ad9a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } } peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } } }
I20260927 02:39:17.439375 8421 ts_tablet_manager.cc:1397] T 524afaa00d134834b85385c9dfa2534d P 97e535c7163e495a87620630469ab388: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:39:17.439422 8429 tablet_replica.cc:442] T 524afaa00d134834b85385c9dfa2534d P d584780e230c4de9b74928bec112ad9a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:39:17.439312 8422 ts_tablet_manager.cc:1428] T 524afaa00d134834b85385c9dfa2534d P d584780e230c4de9b74928bec112ad9a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 02:39:17.439797 8421 raft_consensus.cc:348] T 524afaa00d134834b85385c9dfa2534d P 97e535c7163e495a87620630469ab388 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } } peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } }
I20260927 02:39:17.439903 8421 raft_consensus.cc:374] T 524afaa00d134834b85385c9dfa2534d P 97e535c7163e495a87620630469ab388 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:39:17.439934 8421 raft_consensus.cc:729] T 524afaa00d134834b85385c9dfa2534d P 97e535c7163e495a87620630469ab388 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97e535c7163e495a87620630469ab388, State: Initialized, Role: FOLLOWER
I20260927 02:39:17.440049 8421 consensus_queue.cc:260] T 524afaa00d134834b85385c9dfa2534d P 97e535c7163e495a87620630469ab388 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } } peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } }
I20260927 02:39:17.440294 8421 ts_tablet_manager.cc:1428] T 524afaa00d134834b85385c9dfa2534d P 97e535c7163e495a87620630469ab388: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 02:39:17.440266 8427 tablet_replica.cc:439] T 524afaa00d134834b85385c9dfa2534d P 97e535c7163e495a87620630469ab388: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } } peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } } }
I20260927 02:39:17.440398 8427 tablet_replica.cc:442] T 524afaa00d134834b85385c9dfa2534d P 97e535c7163e495a87620630469ab388: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:39:17.715291 8431 raft_consensus.cc:482] T 524afaa00d134834b85385c9dfa2534d P 6feca25dab664c3ebcd5e224abf3e8a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:39:17.715420 8431 raft_consensus.cc:504] T 524afaa00d134834b85385c9dfa2534d P 6feca25dab664c3ebcd5e224abf3e8a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } } peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } }
I20260927 02:39:17.715816 8431 leader_election.cc:288] T 524afaa00d134834b85385c9dfa2534d P 6feca25dab664c3ebcd5e224abf3e8a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d584780e230c4de9b74928bec112ad9a (127.0.98.1:44417), 97e535c7163e495a87620630469ab388 (127.0.98.3:41159)
I20260927 02:39:17.716243 8347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "524afaa00d134834b85385c9dfa2534d" candidate_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97e535c7163e495a87620630469ab388" is_pre_election: true
I20260927 02:39:17.716259 8067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "524afaa00d134834b85385c9dfa2534d" candidate_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d584780e230c4de9b74928bec112ad9a" is_pre_election: true
I20260927 02:39:17.716377 8347 raft_consensus.cc:2454] T 524afaa00d134834b85385c9dfa2534d P 97e535c7163e495a87620630469ab388 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6feca25dab664c3ebcd5e224abf3e8a7 in term 0.
I20260927 02:39:17.716394 8067 raft_consensus.cc:2454] T 524afaa00d134834b85385c9dfa2534d P d584780e230c4de9b74928bec112ad9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6feca25dab664c3ebcd5e224abf3e8a7 in term 0.
I20260927 02:39:17.716681 8139 leader_election.cc:302] T 524afaa00d134834b85385c9dfa2534d P 6feca25dab664c3ebcd5e224abf3e8a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6feca25dab664c3ebcd5e224abf3e8a7, d584780e230c4de9b74928bec112ad9a; no voters:
I20260927 02:39:17.716846 8431 raft_consensus.cc:2781] T 524afaa00d134834b85385c9dfa2534d P 6feca25dab664c3ebcd5e224abf3e8a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:39:17.716904 8431 raft_consensus.cc:482] T 524afaa00d134834b85385c9dfa2534d P 6feca25dab664c3ebcd5e224abf3e8a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:39:17.716943 8431 raft_consensus.cc:3037] T 524afaa00d134834b85385c9dfa2534d P 6feca25dab664c3ebcd5e224abf3e8a7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:39:17.717978 8431 raft_consensus.cc:504] T 524afaa00d134834b85385c9dfa2534d P 6feca25dab664c3ebcd5e224abf3e8a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } } peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } }
I20260927 02:39:17.718330 8431 leader_election.cc:288] T 524afaa00d134834b85385c9dfa2534d P 6feca25dab664c3ebcd5e224abf3e8a7 [CANDIDATE]: Term 1 election: Requested vote from peers d584780e230c4de9b74928bec112ad9a (127.0.98.1:44417), 97e535c7163e495a87620630469ab388 (127.0.98.3:41159)
I20260927 02:39:17.718616 8067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "524afaa00d134834b85385c9dfa2534d" candidate_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d584780e230c4de9b74928bec112ad9a"
I20260927 02:39:17.718700 8347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "524afaa00d134834b85385c9dfa2534d" candidate_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97e535c7163e495a87620630469ab388"
I20260927 02:39:17.718751 8067 raft_consensus.cc:3037] T 524afaa00d134834b85385c9dfa2534d P d584780e230c4de9b74928bec112ad9a [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:39:17.718807 8347 raft_consensus.cc:3037] T 524afaa00d134834b85385c9dfa2534d P 97e535c7163e495a87620630469ab388 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:39:17.719796 8067 raft_consensus.cc:2454] T 524afaa00d134834b85385c9dfa2534d P d584780e230c4de9b74928bec112ad9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6feca25dab664c3ebcd5e224abf3e8a7 in term 1.
I20260927 02:39:17.719796 8347 raft_consensus.cc:2454] T 524afaa00d134834b85385c9dfa2534d P 97e535c7163e495a87620630469ab388 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6feca25dab664c3ebcd5e224abf3e8a7 in term 1.
I20260927 02:39:17.720091 8138 leader_election.cc:302] T 524afaa00d134834b85385c9dfa2534d P 6feca25dab664c3ebcd5e224abf3e8a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6feca25dab664c3ebcd5e224abf3e8a7, 97e535c7163e495a87620630469ab388; no voters:
I20260927 02:39:17.720252 8431 raft_consensus.cc:2781] T 524afaa00d134834b85385c9dfa2534d P 6feca25dab664c3ebcd5e224abf3e8a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:39:17.720441 8431 raft_consensus.cc:686] T 524afaa00d134834b85385c9dfa2534d P 6feca25dab664c3ebcd5e224abf3e8a7 [term 1 LEADER]: Becoming Leader. State: Replica: 6feca25dab664c3ebcd5e224abf3e8a7, State: Running, Role: LEADER
I20260927 02:39:17.720578 8431 consensus_queue.cc:237] T 524afaa00d134834b85385c9dfa2534d P 6feca25dab664c3ebcd5e224abf3e8a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } } peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } }
I20260927 02:39:17.721172 8464 tablet_replica.cc:439] T 524afaa00d134834b85385c9dfa2534d P 6feca25dab664c3ebcd5e224abf3e8a7: TxnStatusTablet state changed. Reason: New leader 6feca25dab664c3ebcd5e224abf3e8a7. Latest consensus state: current_term: 1 leader_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } } peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } } }
I20260927 02:39:17.721278 8464 tablet_replica.cc:442] T 524afaa00d134834b85385c9dfa2534d P 6feca25dab664c3ebcd5e224abf3e8a7: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:39:17.721547 8466 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 02:39:17.721709 8466 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 02:39:17.722172 7910 catalog_manager.cc:5495] T 524afaa00d134834b85385c9dfa2534d P 6feca25dab664c3ebcd5e224abf3e8a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6feca25dab664c3ebcd5e224abf3e8a7 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } health_report { overall_health: UNKNOWN } } }
I20260927 02:39:17.804476 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:17.800700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45614 (local address 127.0.98.2:39041)
0927 02:39:17.800829 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:17.800832 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:17.800845 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:39:17.800909 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:17.800911 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:17.800955 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:17.801029 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:39:17.801354 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:17.802142 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:17.803360 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:17.803695 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:17.803795 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:17.804053 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:39:17.804129 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:39:17.804186 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:17.804288 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 02:39:17.804340 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 02:39:17.804519 8459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:17.799626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39041 (local address 127.0.0.1:45614)
0927 02:39:17.799823 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:17.799860 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:17.799974 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:17.801124 (+ 1150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:17.801128 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:17.801139 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:39:17.801257 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:17.801262 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:17.802282 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:17.802286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:17.803218 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:17.803229 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:17.803798 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:17.803802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:17.803887 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:17.803932 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:39:17.804197 (+ 265us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:39:17.804213 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 02:39:17.804265 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 02:39:17.804375 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 02:39:17.807631 8067 raft_consensus.cc:1261] T 524afaa00d134834b85385c9dfa2534d P d584780e230c4de9b74928bec112ad9a [term 1 FOLLOWER]: Refusing update from remote peer 6feca25dab664c3ebcd5e224abf3e8a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:39:17.808007 8347 raft_consensus.cc:1261] T 524afaa00d134834b85385c9dfa2534d P 97e535c7163e495a87620630469ab388 [term 1 FOLLOWER]: Refusing update from remote peer 6feca25dab664c3ebcd5e224abf3e8a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:39:17.807999 8431 consensus_queue.cc:1035] T 524afaa00d134834b85385c9dfa2534d P 6feca25dab664c3ebcd5e224abf3e8a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:39:17.808357 8431 consensus_queue.cc:1035] T 524afaa00d134834b85385c9dfa2534d P 6feca25dab664c3ebcd5e224abf3e8a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:39:17.809878 8427 tablet_replica.cc:439] T 524afaa00d134834b85385c9dfa2534d P 97e535c7163e495a87620630469ab388: TxnStatusTablet state changed. Reason: New leader 6feca25dab664c3ebcd5e224abf3e8a7. Latest consensus state: current_term: 1 leader_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } } peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } } }
I20260927 02:39:17.810005 8427 tablet_replica.cc:442] T 524afaa00d134834b85385c9dfa2534d P 97e535c7163e495a87620630469ab388: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:39:17.811002 8429 tablet_replica.cc:439] T 524afaa00d134834b85385c9dfa2534d P d584780e230c4de9b74928bec112ad9a: TxnStatusTablet state changed. Reason: New leader 6feca25dab664c3ebcd5e224abf3e8a7. Latest consensus state: current_term: 1 leader_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } } peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } } }
I20260927 02:39:17.811117 8429 tablet_replica.cc:442] T 524afaa00d134834b85385c9dfa2534d P d584780e230c4de9b74928bec112ad9a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:39:17.812608 8464 tablet_replica.cc:439] T 524afaa00d134834b85385c9dfa2534d P 6feca25dab664c3ebcd5e224abf3e8a7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } } peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } } }
I20260927 02:39:17.812708 8464 tablet_replica.cc:442] T 524afaa00d134834b85385c9dfa2534d P 6feca25dab664c3ebcd5e224abf3e8a7: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:39:17.813102 8464 tablet_replica.cc:439] T 524afaa00d134834b85385c9dfa2534d P 6feca25dab664c3ebcd5e224abf3e8a7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } } peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } } }
I20260927 02:39:17.813192 8464 tablet_replica.cc:442] T 524afaa00d134834b85385c9dfa2534d P 6feca25dab664c3ebcd5e224abf3e8a7: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:39:17.813462 8427 tablet_replica.cc:439] T 524afaa00d134834b85385c9dfa2534d P 97e535c7163e495a87620630469ab388: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } } peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } } }
I20260927 02:39:17.813556 8427 tablet_replica.cc:442] T 524afaa00d134834b85385c9dfa2534d P 97e535c7163e495a87620630469ab388: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:39:17.814123 8429 tablet_replica.cc:439] T 524afaa00d134834b85385c9dfa2534d P d584780e230c4de9b74928bec112ad9a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } } peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } } }
I20260927 02:39:17.814221 8429 tablet_replica.cc:442] T 524afaa00d134834b85385c9dfa2534d P d584780e230c4de9b74928bec112ad9a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:39:17.825635 8479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:17.822154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54360 (local address 127.0.98.3:41159)
0927 02:39:17.822360 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:17.822365 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:17.822380 (+ 15us) server_negotiation.cc:407] Connection header received
0927 02:39:17.822469 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:17.822472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:17.822532 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:17.822642 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:39:17.822914 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:17.823432 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:17.824246 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:17.824486 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:17.824570 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:17.824819 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:39:17.825255 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:39:17.825356 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:17.825438 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 02:39:17.825492 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":66,"threads_started":1}
I20260927 02:39:17.836014 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:17.832397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39041 (local address 127.0.98.3:59501)
0927 02:39:17.832657 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:17.832671 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:17.832754 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:17.833010 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:17.833012 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:17.833021 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:39:17.833108 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:17.833112 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:17.834109 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:17.834112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:17.834835 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:17.834844 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:17.835396 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:17.835399 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:17.835459 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:17.835506 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:39:17.835766 (+ 260us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:39:17.835770 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:39:17.835812 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 02:39:17.835875 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":86,"threads_started":1}
I20260927 02:39:17.836009 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:17.832519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59501 (local address 127.0.98.2:39041)
0927 02:39:17.832637 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:17.832642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:17.832749 (+ 107us) server_negotiation.cc:407] Connection header received
0927 02:39:17.832848 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:17.832852 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:17.832909 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:17.832993 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:39:17.833214 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:17.833984 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:17.834958 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:17.835293 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:17.835399 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:17.835621 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:39:17.835698 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:39:17.835762 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:17.835825 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 02:39:17.835876 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 02:39:17.842850 8207 raft_consensus.cc:1261] T ac045a4e76dc463e82b3c1c19458fb8a P 6feca25dab664c3ebcd5e224abf3e8a7 [term 1 FOLLOWER]: Refusing update from remote peer 97e535c7163e495a87620630469ab388: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:39:17.843297 8067 raft_consensus.cc:1261] T ac045a4e76dc463e82b3c1c19458fb8a P d584780e230c4de9b74928bec112ad9a [term 1 FOLLOWER]: Refusing update from remote peer 97e535c7163e495a87620630469ab388: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:39:17.843364 8452 consensus_queue.cc:1035] T ac045a4e76dc463e82b3c1c19458fb8a P 97e535c7163e495a87620630469ab388 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:39:17.843636 8427 consensus_queue.cc:1035] T ac045a4e76dc463e82b3c1c19458fb8a P 97e535c7163e495a87620630469ab388 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:39:17.863757 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:17.860356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41159 (local address 127.0.98.2:60745)
0927 02:39:17.860658 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:17.860673 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:17.860770 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:17.860967 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:17.860969 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:17.860976 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:39:17.861084 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:17.861088 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:17.861734 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:17.861738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:17.862612 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:17.862622 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:17.863165 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:17.863168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:17.863230 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:17.863277 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:39:17.863540 (+ 263us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:39:17.863544 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:39:17.863583 (+ 39us) client_negotiation.cc:240] Negotiation successful
0927 02:39:17.863647 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":119,"threads_started":1}
I20260927 02:39:17.863757 8479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:17.860404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60745 (local address 127.0.98.3:41159)
0927 02:39:17.860509 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:17.860513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:17.860739 (+ 226us) server_negotiation.cc:407] Connection header received
0927 02:39:17.860848 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:17.860850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:17.860894 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:17.860950 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:39:17.861149 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:17.861626 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:17.862735 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:17.863049 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:17.863146 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:17.863383 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:39:17.863469 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:39:17.863539 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:17.863594 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 02:39:17.863641 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 02:39:17.883803 7908 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:39:17.883985 7908 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:39:17.885978 7908 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c03c951086c4af5b892870108fd8990: Sending DeleteTablet for 3 replicas of tablet ac045a4e76dc463e82b3c1c19458fb8a
I20260927 02:39:17.886538 8047 tablet_service.cc:1514] Processing DeleteTablet for tablet ac045a4e76dc463e82b3c1c19458fb8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:39:17 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49626
I20260927 02:39:17.886592 8187 tablet_service.cc:1514] Processing DeleteTablet for tablet ac045a4e76dc463e82b3c1c19458fb8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:39:17 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45594
I20260927 02:39:17.886865 8327 tablet_service.cc:1514] Processing DeleteTablet for tablet ac045a4e76dc463e82b3c1c19458fb8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:39:17 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54358
I20260927 02:39:17.887085 8492 tablet_replica.cc:330] T ac045a4e76dc463e82b3c1c19458fb8a P 6feca25dab664c3ebcd5e224abf3e8a7: stopping tablet replica
I20260927 02:39:17.887209 8493 tablet_replica.cc:330] T ac045a4e76dc463e82b3c1c19458fb8a P 97e535c7163e495a87620630469ab388: stopping tablet replica
I20260927 02:39:17.887303 8492 raft_consensus.cc:2229] T ac045a4e76dc463e82b3c1c19458fb8a P 6feca25dab664c3ebcd5e224abf3e8a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:39:17.887333 8493 raft_consensus.cc:2229] T ac045a4e76dc463e82b3c1c19458fb8a P 97e535c7163e495a87620630469ab388 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:39:17.887509 8492 raft_consensus.cc:2258] T ac045a4e76dc463e82b3c1c19458fb8a P 6feca25dab664c3ebcd5e224abf3e8a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:39:17.887612 8493 raft_consensus.cc:2258] T ac045a4e76dc463e82b3c1c19458fb8a P 97e535c7163e495a87620630469ab388 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:39:17.888275 8493 ts_tablet_manager.cc:1905] T ac045a4e76dc463e82b3c1c19458fb8a P 97e535c7163e495a87620630469ab388: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:39:17.888275 8492 ts_tablet_manager.cc:1905] T ac045a4e76dc463e82b3c1c19458fb8a P 6feca25dab664c3ebcd5e224abf3e8a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:39:17.891175 8493 ts_tablet_manager.cc:1918] T ac045a4e76dc463e82b3c1c19458fb8a P 97e535c7163e495a87620630469ab388: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:39:17.891256 8493 log.cc:1196] T ac045a4e76dc463e82b3c1c19458fb8a P 97e535c7163e495a87620630469ab388: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-2/wal/wals/ac045a4e76dc463e82b3c1c19458fb8a
I20260927 02:39:17.891480 8491 tablet_replica.cc:330] T ac045a4e76dc463e82b3c1c19458fb8a P d584780e230c4de9b74928bec112ad9a: stopping tablet replica
I20260927 02:39:17.891510 8493 ts_tablet_manager.cc:1939] T ac045a4e76dc463e82b3c1c19458fb8a P 97e535c7163e495a87620630469ab388: Deleting consensus metadata
I20260927 02:39:17.891683 8491 raft_consensus.cc:2229] T ac045a4e76dc463e82b3c1c19458fb8a P d584780e230c4de9b74928bec112ad9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:39:17.891842 8491 raft_consensus.cc:2258] T ac045a4e76dc463e82b3c1c19458fb8a P d584780e230c4de9b74928bec112ad9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:39:17.891225 8492 ts_tablet_manager.cc:1918] T ac045a4e76dc463e82b3c1c19458fb8a P 6feca25dab664c3ebcd5e224abf3e8a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:39:17.892100 8492 log.cc:1196] T ac045a4e76dc463e82b3c1c19458fb8a P 6feca25dab664c3ebcd5e224abf3e8a7: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-1/wal/wals/ac045a4e76dc463e82b3c1c19458fb8a
I20260927 02:39:17.892195 7894 catalog_manager.cc:4841] TS 97e535c7163e495a87620630469ab388 (127.0.98.3:41159): tablet ac045a4e76dc463e82b3c1c19458fb8a (table test-table [id=7c0e7bf0b98947e08d1402fc33967ece]) successfully deleted
I20260927 02:39:17.892400 8492 ts_tablet_manager.cc:1939] T ac045a4e76dc463e82b3c1c19458fb8a P 6feca25dab664c3ebcd5e224abf3e8a7: Deleting consensus metadata
I20260927 02:39:17.892575 8491 ts_tablet_manager.cc:1905] T ac045a4e76dc463e82b3c1c19458fb8a P d584780e230c4de9b74928bec112ad9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:39:17.893061 7897 catalog_manager.cc:4841] TS 6feca25dab664c3ebcd5e224abf3e8a7 (127.0.98.2:39041): tablet ac045a4e76dc463e82b3c1c19458fb8a (table test-table [id=7c0e7bf0b98947e08d1402fc33967ece]) successfully deleted
I20260927 02:39:17.895335 8491 ts_tablet_manager.cc:1918] T ac045a4e76dc463e82b3c1c19458fb8a P d584780e230c4de9b74928bec112ad9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:39:17.895402 8491 log.cc:1196] T ac045a4e76dc463e82b3c1c19458fb8a P d584780e230c4de9b74928bec112ad9a: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-0/wal/wals/ac045a4e76dc463e82b3c1c19458fb8a
I20260927 02:39:17.895629 8491 ts_tablet_manager.cc:1939] T ac045a4e76dc463e82b3c1c19458fb8a P d584780e230c4de9b74928bec112ad9a: Deleting consensus metadata
I20260927 02:39:17.896301 7895 catalog_manager.cc:4841] TS d584780e230c4de9b74928bec112ad9a (127.0.98.1:44417): tablet ac045a4e76dc463e82b3c1c19458fb8a (table test-table [id=7c0e7bf0b98947e08d1402fc33967ece]) successfully deleted
Sep 27 02:39:17 dist-test-slave-zv8z krb5kdc[7865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 02:39:17 dist-test-slave-zv8z krb5kdc[7865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:17 dist-test-slave-zv8z krb5kdc[7865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 02:39:17.917333 8460 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:17.910968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35704 (local address 127.0.98.62:37181)
0927 02:39:17.911104 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:17.911109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:17.911245 (+ 136us) server_negotiation.cc:407] Connection header received
0927 02:39:17.911357 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:17.911360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:17.911408 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:17.911495 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:17.912116 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:17.912933 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:17.913817 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:17.914170 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:17.914289 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:17.917211 (+ 2922us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35704: BlockingRecv error: recv got EOF from 127.0.0.1:35704 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 27 02:39:17 dist-test-slave-zv8z krb5kdc[7865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:39:17.926122 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:17.918666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35708 (local address 127.0.98.62:37181)
0927 02:39:17.918766 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:17.918770 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:17.918902 (+ 132us) server_negotiation.cc:407] Connection header received
0927 02:39:17.919054 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:17.919057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:17.919095 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:17.919156 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:17.919703 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:17.920173 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:17.920911 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:17.921109 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:17.921190 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:17.923479 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:17.923483 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:17.923486 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:17.923512 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:17.924726 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:17.925080 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:17.925084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:17.925085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:17.925152 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:17.925379 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:17.925382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:17.925384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:17.925546 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:17.925659 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:17.925867 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 02:39:17.925983 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 02:39:17.928589 7908 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35708:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:39:17.928995 7908 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:39:17.933357 8047 tablet_service.cc:1467] Processing CreateTablet for tablet 2556c7ba4dde4cb0b9dd8684f0dda4e9 (DEFAULT_TABLE table=test-table [id=14028b106506447f8ddc5d6cef5aa454]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:39:17.933647 8047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2556c7ba4dde4cb0b9dd8684f0dda4e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:39:17.933599 8327 tablet_service.cc:1467] Processing CreateTablet for tablet 2556c7ba4dde4cb0b9dd8684f0dda4e9 (DEFAULT_TABLE table=test-table [id=14028b106506447f8ddc5d6cef5aa454]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:39:17.933619 8187 tablet_service.cc:1467] Processing CreateTablet for tablet 2556c7ba4dde4cb0b9dd8684f0dda4e9 (DEFAULT_TABLE table=test-table [id=14028b106506447f8ddc5d6cef5aa454]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:39:17.933840 8327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2556c7ba4dde4cb0b9dd8684f0dda4e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:39:17.933881 8187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2556c7ba4dde4cb0b9dd8684f0dda4e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:39:17.936071 8422 tablet_bootstrap.cc:492] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P d584780e230c4de9b74928bec112ad9a: Bootstrap starting.
I20260927 02:39:17.936084 8421 tablet_bootstrap.cc:492] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P 97e535c7163e495a87620630469ab388: Bootstrap starting.
I20260927 02:39:17.936151 8423 tablet_bootstrap.cc:492] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P 6feca25dab664c3ebcd5e224abf3e8a7: Bootstrap starting.
I20260927 02:39:17.936942 8422 tablet_bootstrap.cc:654] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P d584780e230c4de9b74928bec112ad9a: Neither blocks nor log segments found. Creating new log.
I20260927 02:39:17.936942 8421 tablet_bootstrap.cc:654] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P 97e535c7163e495a87620630469ab388: Neither blocks nor log segments found. Creating new log.
I20260927 02:39:17.936945 8423 tablet_bootstrap.cc:654] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P 6feca25dab664c3ebcd5e224abf3e8a7: Neither blocks nor log segments found. Creating new log.
I20260927 02:39:17.938069 8421 tablet_bootstrap.cc:492] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P 97e535c7163e495a87620630469ab388: No bootstrap required, opened a new log
I20260927 02:39:17.938158 8421 ts_tablet_manager.cc:1397] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P 97e535c7163e495a87620630469ab388: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:39:17.938434 8423 tablet_bootstrap.cc:492] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P 6feca25dab664c3ebcd5e224abf3e8a7: No bootstrap required, opened a new log
I20260927 02:39:17.938513 8423 ts_tablet_manager.cc:1397] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P 6feca25dab664c3ebcd5e224abf3e8a7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:39:17.938658 8421 raft_consensus.cc:348] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P 97e535c7163e495a87620630469ab388 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } } peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } }
I20260927 02:39:17.938787 8421 raft_consensus.cc:374] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P 97e535c7163e495a87620630469ab388 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:39:17.938824 8421 raft_consensus.cc:729] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P 97e535c7163e495a87620630469ab388 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97e535c7163e495a87620630469ab388, State: Initialized, Role: FOLLOWER
I20260927 02:39:17.938916 8423 raft_consensus.cc:348] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P 6feca25dab664c3ebcd5e224abf3e8a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } } peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } }
I20260927 02:39:17.938941 8421 consensus_queue.cc:260] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P 97e535c7163e495a87620630469ab388 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } } peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } }
I20260927 02:39:17.939020 8423 raft_consensus.cc:374] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P 6feca25dab664c3ebcd5e224abf3e8a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:39:17.939051 8423 raft_consensus.cc:729] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P 6feca25dab664c3ebcd5e224abf3e8a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6feca25dab664c3ebcd5e224abf3e8a7, State: Initialized, Role: FOLLOWER
I20260927 02:39:17.939157 8423 consensus_queue.cc:260] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P 6feca25dab664c3ebcd5e224abf3e8a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } } peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } }
I20260927 02:39:17.939205 8421 ts_tablet_manager.cc:1428] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P 97e535c7163e495a87620630469ab388: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 02:39:17.939399 8423 ts_tablet_manager.cc:1428] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P 6feca25dab664c3ebcd5e224abf3e8a7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:39:17.941720 8422 tablet_bootstrap.cc:492] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P d584780e230c4de9b74928bec112ad9a: No bootstrap required, opened a new log
I20260927 02:39:17.941802 8422 ts_tablet_manager.cc:1397] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P d584780e230c4de9b74928bec112ad9a: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260927 02:39:17.942224 8422 raft_consensus.cc:348] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P d584780e230c4de9b74928bec112ad9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } } peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } }
I20260927 02:39:17.942365 8422 raft_consensus.cc:374] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P d584780e230c4de9b74928bec112ad9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:39:17.942404 8422 raft_consensus.cc:729] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P d584780e230c4de9b74928bec112ad9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d584780e230c4de9b74928bec112ad9a, State: Initialized, Role: FOLLOWER
I20260927 02:39:17.942512 8422 consensus_queue.cc:260] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P d584780e230c4de9b74928bec112ad9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } } peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } }
I20260927 02:39:17.942757 8422 ts_tablet_manager.cc:1428] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P d584780e230c4de9b74928bec112ad9a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:39:18.217851 8488 raft_consensus.cc:482] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P 97e535c7163e495a87620630469ab388 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:39:18.217981 8488 raft_consensus.cc:504] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P 97e535c7163e495a87620630469ab388 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } } peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } }
I20260927 02:39:18.218394 8488 leader_election.cc:288] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P 97e535c7163e495a87620630469ab388 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d584780e230c4de9b74928bec112ad9a (127.0.98.1:44417), 6feca25dab664c3ebcd5e224abf3e8a7 (127.0.98.2:39041)
I20260927 02:39:18.218880 8207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2556c7ba4dde4cb0b9dd8684f0dda4e9" candidate_uuid: "97e535c7163e495a87620630469ab388" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" is_pre_election: true
I20260927 02:39:18.218904 8065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2556c7ba4dde4cb0b9dd8684f0dda4e9" candidate_uuid: "97e535c7163e495a87620630469ab388" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d584780e230c4de9b74928bec112ad9a" is_pre_election: true
I20260927 02:39:18.219038 8065 raft_consensus.cc:2454] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P d584780e230c4de9b74928bec112ad9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97e535c7163e495a87620630469ab388 in term 0.
I20260927 02:39:18.219036 8207 raft_consensus.cc:2454] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P 6feca25dab664c3ebcd5e224abf3e8a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97e535c7163e495a87620630469ab388 in term 0.
I20260927 02:39:18.219347 8281 leader_election.cc:302] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P 97e535c7163e495a87620630469ab388 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6feca25dab664c3ebcd5e224abf3e8a7, 97e535c7163e495a87620630469ab388; no voters:
I20260927 02:39:18.219563 8488 raft_consensus.cc:2781] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P 97e535c7163e495a87620630469ab388 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:39:18.219622 8488 raft_consensus.cc:482] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P 97e535c7163e495a87620630469ab388 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:39:18.219648 8488 raft_consensus.cc:3037] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P 97e535c7163e495a87620630469ab388 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:39:18.220672 8488 raft_consensus.cc:504] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P 97e535c7163e495a87620630469ab388 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } } peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } }
I20260927 02:39:18.220983 8488 leader_election.cc:288] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P 97e535c7163e495a87620630469ab388 [CANDIDATE]: Term 1 election: Requested vote from peers d584780e230c4de9b74928bec112ad9a (127.0.98.1:44417), 6feca25dab664c3ebcd5e224abf3e8a7 (127.0.98.2:39041)
I20260927 02:39:18.221367 8065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2556c7ba4dde4cb0b9dd8684f0dda4e9" candidate_uuid: "97e535c7163e495a87620630469ab388" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d584780e230c4de9b74928bec112ad9a"
I20260927 02:39:18.221367 8207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2556c7ba4dde4cb0b9dd8684f0dda4e9" candidate_uuid: "97e535c7163e495a87620630469ab388" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6feca25dab664c3ebcd5e224abf3e8a7"
I20260927 02:39:18.221503 8207 raft_consensus.cc:3037] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P 6feca25dab664c3ebcd5e224abf3e8a7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:39:18.221503 8065 raft_consensus.cc:3037] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P d584780e230c4de9b74928bec112ad9a [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:39:18.222695 8207 raft_consensus.cc:2454] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P 6feca25dab664c3ebcd5e224abf3e8a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97e535c7163e495a87620630469ab388 in term 1.
I20260927 02:39:18.222697 8065 raft_consensus.cc:2454] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P d584780e230c4de9b74928bec112ad9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97e535c7163e495a87620630469ab388 in term 1.
I20260927 02:39:18.222972 8279 leader_election.cc:302] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P 97e535c7163e495a87620630469ab388 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97e535c7163e495a87620630469ab388, d584780e230c4de9b74928bec112ad9a; no voters:
I20260927 02:39:18.223161 8488 raft_consensus.cc:2781] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P 97e535c7163e495a87620630469ab388 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:39:18.223284 8488 raft_consensus.cc:686] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P 97e535c7163e495a87620630469ab388 [term 1 LEADER]: Becoming Leader. State: Replica: 97e535c7163e495a87620630469ab388, State: Running, Role: LEADER
I20260927 02:39:18.223416 8488 consensus_queue.cc:237] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P 97e535c7163e495a87620630469ab388 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } } peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } }
I20260927 02:39:18.224956 7908 catalog_manager.cc:5495] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P 97e535c7163e495a87620630469ab388 reported cstate change: term changed from 0 to 1, leader changed from <none> to 97e535c7163e495a87620630469ab388 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "97e535c7163e495a87620630469ab388" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } health_report { overall_health: HEALTHY } } }
I20260927 02:39:18.235219 8479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:18.232237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54376 (local address 127.0.98.3:41159)
0927 02:39:18.232359 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:18.232363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:18.232377 (+ 14us) server_negotiation.cc:407] Connection header received
0927 02:39:18.232470 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:18.232473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:18.232516 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:18.232579 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:39:18.232829 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:18.233353 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:18.234222 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:18.234499 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:18.234586 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:18.234832 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:39:18.234901 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:39:18.234956 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:18.235047 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 02:39:18.235095 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 02:39:18.236850 8065 raft_consensus.cc:1261] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P d584780e230c4de9b74928bec112ad9a [term 1 FOLLOWER]: Refusing update from remote peer 97e535c7163e495a87620630469ab388: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:39:18.236867 8207 raft_consensus.cc:1261] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P 6feca25dab664c3ebcd5e224abf3e8a7 [term 1 FOLLOWER]: Refusing update from remote peer 97e535c7163e495a87620630469ab388: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:39:18.237228 8427 consensus_queue.cc:1035] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P 97e535c7163e495a87620630469ab388 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:39:18.237375 8488 consensus_queue.cc:1035] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P 97e535c7163e495a87620630469ab388 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:39:18.244282 7909 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:39:18.244396 7909 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:39:18.246238 7909 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c03c951086c4af5b892870108fd8990: Sending DeleteTablet for 3 replicas of tablet 2556c7ba4dde4cb0b9dd8684f0dda4e9
I20260927 02:39:18.246786 8047 tablet_service.cc:1514] Processing DeleteTablet for tablet 2556c7ba4dde4cb0b9dd8684f0dda4e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:39:18 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49626
I20260927 02:39:18.246897 8187 tablet_service.cc:1514] Processing DeleteTablet for tablet 2556c7ba4dde4cb0b9dd8684f0dda4e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:39:18 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45594
I20260927 02:39:18.247002 8327 tablet_service.cc:1514] Processing DeleteTablet for tablet 2556c7ba4dde4cb0b9dd8684f0dda4e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:39:18 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54358
I20260927 02:39:18.247061 8492 tablet_replica.cc:330] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P 6feca25dab664c3ebcd5e224abf3e8a7: stopping tablet replica
I20260927 02:39:18.247159 8493 tablet_replica.cc:330] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P 97e535c7163e495a87620630469ab388: stopping tablet replica
I20260927 02:39:18.247179 8491 tablet_replica.cc:330] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P d584780e230c4de9b74928bec112ad9a: stopping tablet replica
I20260927 02:39:18.247203 8492 raft_consensus.cc:2229] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P 6feca25dab664c3ebcd5e224abf3e8a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:39:18.247252 8493 raft_consensus.cc:2229] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P 97e535c7163e495a87620630469ab388 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:39:18.247284 8491 raft_consensus.cc:2229] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P d584780e230c4de9b74928bec112ad9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:39:18.247318 8492 raft_consensus.cc:2258] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P 6feca25dab664c3ebcd5e224abf3e8a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:39:18.247403 8491 raft_consensus.cc:2258] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P d584780e230c4de9b74928bec112ad9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:39:18.247412 8493 raft_consensus.cc:2258] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P 97e535c7163e495a87620630469ab388 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:39:18.247560 7909 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35708:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:39:18.247927 7909 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:39:18.248028 8493 ts_tablet_manager.cc:1905] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P 97e535c7163e495a87620630469ab388: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:39:18.248119 8492 ts_tablet_manager.cc:1905] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P 6feca25dab664c3ebcd5e224abf3e8a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:39:18.248126 8491 ts_tablet_manager.cc:1905] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P d584780e230c4de9b74928bec112ad9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:39:18.250121 8493 ts_tablet_manager.cc:1918] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P 97e535c7163e495a87620630469ab388: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:39:18.250190 8493 log.cc:1196] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P 97e535c7163e495a87620630469ab388: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-2/wal/wals/2556c7ba4dde4cb0b9dd8684f0dda4e9
I20260927 02:39:18.250416 8492 ts_tablet_manager.cc:1918] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P 6feca25dab664c3ebcd5e224abf3e8a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:39:18.250416 8491 ts_tablet_manager.cc:1918] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P d584780e230c4de9b74928bec112ad9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:39:18.250453 8493 ts_tablet_manager.cc:1939] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P 97e535c7163e495a87620630469ab388: Deleting consensus metadata
I20260927 02:39:18.250491 8491 log.cc:1196] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P d584780e230c4de9b74928bec112ad9a: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-0/wal/wals/2556c7ba4dde4cb0b9dd8684f0dda4e9
I20260927 02:39:18.250499 8492 log.cc:1196] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P 6feca25dab664c3ebcd5e224abf3e8a7: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-1/wal/wals/2556c7ba4dde4cb0b9dd8684f0dda4e9
I20260927 02:39:18.250733 8492 ts_tablet_manager.cc:1939] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P 6feca25dab664c3ebcd5e224abf3e8a7: Deleting consensus metadata
I20260927 02:39:18.250733 8491 ts_tablet_manager.cc:1939] T 2556c7ba4dde4cb0b9dd8684f0dda4e9 P d584780e230c4de9b74928bec112ad9a: Deleting consensus metadata
I20260927 02:39:18.251116 7894 catalog_manager.cc:4841] TS 97e535c7163e495a87620630469ab388 (127.0.98.3:41159): tablet 2556c7ba4dde4cb0b9dd8684f0dda4e9 (table test-table [id=14028b106506447f8ddc5d6cef5aa454]) successfully deleted
I20260927 02:39:18.251307 7895 catalog_manager.cc:4841] TS d584780e230c4de9b74928bec112ad9a (127.0.98.1:44417): tablet 2556c7ba4dde4cb0b9dd8684f0dda4e9 (table test-table [id=14028b106506447f8ddc5d6cef5aa454]) successfully deleted
I20260927 02:39:18.251823 7897 catalog_manager.cc:4841] TS 6feca25dab664c3ebcd5e224abf3e8a7 (127.0.98.2:39041): tablet 2556c7ba4dde4cb0b9dd8684f0dda4e9 (table test-table [id=14028b106506447f8ddc5d6cef5aa454]) successfully deleted
I20260927 02:39:18.254895 8047 tablet_service.cc:1467] Processing CreateTablet for tablet 22b87493dd324d2f947a14bfba1074d2 (DEFAULT_TABLE table=test-table [id=d53440acec204333948422006c747ae1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:39:18.254916 8327 tablet_service.cc:1467] Processing CreateTablet for tablet 22b87493dd324d2f947a14bfba1074d2 (DEFAULT_TABLE table=test-table [id=d53440acec204333948422006c747ae1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:39:18.255177 8047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22b87493dd324d2f947a14bfba1074d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:39:18.255177 8327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22b87493dd324d2f947a14bfba1074d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:39:18.256934 8421 tablet_bootstrap.cc:492] T 22b87493dd324d2f947a14bfba1074d2 P 97e535c7163e495a87620630469ab388: Bootstrap starting.
I20260927 02:39:18.256944 8422 tablet_bootstrap.cc:492] T 22b87493dd324d2f947a14bfba1074d2 P d584780e230c4de9b74928bec112ad9a: Bootstrap starting.
I20260927 02:39:18.257107 8187 tablet_service.cc:1467] Processing CreateTablet for tablet 22b87493dd324d2f947a14bfba1074d2 (DEFAULT_TABLE table=test-table [id=d53440acec204333948422006c747ae1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:39:18.257341 8187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22b87493dd324d2f947a14bfba1074d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:39:18.257701 8421 tablet_bootstrap.cc:654] T 22b87493dd324d2f947a14bfba1074d2 P 97e535c7163e495a87620630469ab388: Neither blocks nor log segments found. Creating new log.
I20260927 02:39:18.257975 8422 tablet_bootstrap.cc:654] T 22b87493dd324d2f947a14bfba1074d2 P d584780e230c4de9b74928bec112ad9a: Neither blocks nor log segments found. Creating new log.
I20260927 02:39:18.259107 8422 tablet_bootstrap.cc:492] T 22b87493dd324d2f947a14bfba1074d2 P d584780e230c4de9b74928bec112ad9a: No bootstrap required, opened a new log
I20260927 02:39:18.259198 8422 ts_tablet_manager.cc:1397] T 22b87493dd324d2f947a14bfba1074d2 P d584780e230c4de9b74928bec112ad9a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:39:18.259577 8423 tablet_bootstrap.cc:492] T 22b87493dd324d2f947a14bfba1074d2 P 6feca25dab664c3ebcd5e224abf3e8a7: Bootstrap starting.
I20260927 02:39:18.259648 8422 raft_consensus.cc:348] T 22b87493dd324d2f947a14bfba1074d2 P d584780e230c4de9b74928bec112ad9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } } peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } }
I20260927 02:39:18.259752 8422 raft_consensus.cc:374] T 22b87493dd324d2f947a14bfba1074d2 P d584780e230c4de9b74928bec112ad9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:39:18.259785 8422 raft_consensus.cc:729] T 22b87493dd324d2f947a14bfba1074d2 P d584780e230c4de9b74928bec112ad9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d584780e230c4de9b74928bec112ad9a, State: Initialized, Role: FOLLOWER
I20260927 02:39:18.259893 8422 consensus_queue.cc:260] T 22b87493dd324d2f947a14bfba1074d2 P d584780e230c4de9b74928bec112ad9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } } peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } }
I20260927 02:39:18.260157 8422 ts_tablet_manager.cc:1428] T 22b87493dd324d2f947a14bfba1074d2 P d584780e230c4de9b74928bec112ad9a: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260927 02:39:18.260298 8423 tablet_bootstrap.cc:654] T 22b87493dd324d2f947a14bfba1074d2 P 6feca25dab664c3ebcd5e224abf3e8a7: Neither blocks nor log segments found. Creating new log.
I20260927 02:39:18.261232 8423 tablet_bootstrap.cc:492] T 22b87493dd324d2f947a14bfba1074d2 P 6feca25dab664c3ebcd5e224abf3e8a7: No bootstrap required, opened a new log
I20260927 02:39:18.261317 8423 ts_tablet_manager.cc:1397] T 22b87493dd324d2f947a14bfba1074d2 P 6feca25dab664c3ebcd5e224abf3e8a7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:39:18.261646 8423 raft_consensus.cc:348] T 22b87493dd324d2f947a14bfba1074d2 P 6feca25dab664c3ebcd5e224abf3e8a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } } peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } }
I20260927 02:39:18.261731 8423 raft_consensus.cc:374] T 22b87493dd324d2f947a14bfba1074d2 P 6feca25dab664c3ebcd5e224abf3e8a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:39:18.261758 8423 raft_consensus.cc:729] T 22b87493dd324d2f947a14bfba1074d2 P 6feca25dab664c3ebcd5e224abf3e8a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6feca25dab664c3ebcd5e224abf3e8a7, State: Initialized, Role: FOLLOWER
I20260927 02:39:18.261874 8423 consensus_queue.cc:260] T 22b87493dd324d2f947a14bfba1074d2 P 6feca25dab664c3ebcd5e224abf3e8a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } } peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } }
I20260927 02:39:18.262107 8423 ts_tablet_manager.cc:1428] T 22b87493dd324d2f947a14bfba1074d2 P 6feca25dab664c3ebcd5e224abf3e8a7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 02:39:18.266767 8421 tablet_bootstrap.cc:492] T 22b87493dd324d2f947a14bfba1074d2 P 97e535c7163e495a87620630469ab388: No bootstrap required, opened a new log
I20260927 02:39:18.266852 8421 ts_tablet_manager.cc:1397] T 22b87493dd324d2f947a14bfba1074d2 P 97e535c7163e495a87620630469ab388: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.001s
I20260927 02:39:18.267206 8421 raft_consensus.cc:348] T 22b87493dd324d2f947a14bfba1074d2 P 97e535c7163e495a87620630469ab388 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } } peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } }
I20260927 02:39:18.267300 8421 raft_consensus.cc:374] T 22b87493dd324d2f947a14bfba1074d2 P 97e535c7163e495a87620630469ab388 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:39:18.267329 8421 raft_consensus.cc:729] T 22b87493dd324d2f947a14bfba1074d2 P 97e535c7163e495a87620630469ab388 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97e535c7163e495a87620630469ab388, State: Initialized, Role: FOLLOWER
I20260927 02:39:18.267457 8421 consensus_queue.cc:260] T 22b87493dd324d2f947a14bfba1074d2 P 97e535c7163e495a87620630469ab388 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } } peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } }
I20260927 02:39:18.267704 8421 ts_tablet_manager.cc:1428] T 22b87493dd324d2f947a14bfba1074d2 P 97e535c7163e495a87620630469ab388: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 02:39:18.298647 8429 raft_consensus.cc:482] T 22b87493dd324d2f947a14bfba1074d2 P d584780e230c4de9b74928bec112ad9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:39:18.298785 8429 raft_consensus.cc:504] T 22b87493dd324d2f947a14bfba1074d2 P d584780e230c4de9b74928bec112ad9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } } peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } }
I20260927 02:39:18.299898 8429 leader_election.cc:288] T 22b87493dd324d2f947a14bfba1074d2 P d584780e230c4de9b74928bec112ad9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6feca25dab664c3ebcd5e224abf3e8a7 (127.0.98.2:39041), 97e535c7163e495a87620630469ab388 (127.0.98.3:41159)
I20260927 02:39:18.303435 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:18.300225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60081 (local address 127.0.98.2:39041)
0927 02:39:18.300369 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:18.300374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:18.300392 (+ 18us) server_negotiation.cc:407] Connection header received
0927 02:39:18.300579 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:18.300581 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:18.300622 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:18.300699 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:39:18.300979 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:18.301505 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:18.302567 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:18.302949 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:18.303039 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:18.303108 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:18.303255 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 02:39:18.303318 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 02:39:18.303488 8479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:18.300372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41963 (local address 127.0.98.3:41159)
0927 02:39:18.300486 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:18.300489 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:18.300663 (+ 174us) server_negotiation.cc:407] Connection header received
0927 02:39:18.300756 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:18.300759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:18.300802 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:18.300880 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:39:18.301101 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:18.301581 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:18.302638 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:18.303014 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:18.303103 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:18.303159 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:18.303315 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 02:39:18.303378 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 02:39:18.303469 8516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:18.300083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39041 (local address 127.0.98.1:60081)
0927 02:39:18.300332 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:18.300347 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:18.300442 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:18.300718 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:18.300721 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:18.300736 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:39:18.300881 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:18.300889 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:18.301612 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:18.301616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:18.302461 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:18.302470 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:18.303069 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:18.303073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:18.303155 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:18.303172 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 02:39:18.303235 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 02:39:18.303299 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":72,"threads_started":1}
I20260927 02:39:18.303504 8517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:18.300136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41159 (local address 127.0.98.1:41963)
0927 02:39:18.300584 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:18.300596 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:18.300681 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:18.300905 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:18.300908 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:18.300928 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:39:18.301015 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:18.301019 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:18.301697 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:18.301700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:18.302536 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:18.302545 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:18.303120 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:18.303123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:18.303211 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:18.303232 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 02:39:18.303293 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 02:39:18.303354 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":97,"threads_started":1}
I20260927 02:39:18.303993 8207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b87493dd324d2f947a14bfba1074d2" candidate_uuid: "d584780e230c4de9b74928bec112ad9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" is_pre_election: true
I20260927 02:39:18.304131 8207 raft_consensus.cc:2454] T 22b87493dd324d2f947a14bfba1074d2 P 6feca25dab664c3ebcd5e224abf3e8a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d584780e230c4de9b74928bec112ad9a in term 0.
I20260927 02:39:18.304171 8347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b87493dd324d2f947a14bfba1074d2" candidate_uuid: "d584780e230c4de9b74928bec112ad9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97e535c7163e495a87620630469ab388" is_pre_election: true
I20260927 02:39:18.304301 8347 raft_consensus.cc:2454] T 22b87493dd324d2f947a14bfba1074d2 P 97e535c7163e495a87620630469ab388 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d584780e230c4de9b74928bec112ad9a in term 0.
I20260927 02:39:18.304452 8001 leader_election.cc:302] T 22b87493dd324d2f947a14bfba1074d2 P d584780e230c4de9b74928bec112ad9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6feca25dab664c3ebcd5e224abf3e8a7, d584780e230c4de9b74928bec112ad9a; no voters:
I20260927 02:39:18.304718 8429 raft_consensus.cc:2781] T 22b87493dd324d2f947a14bfba1074d2 P d584780e230c4de9b74928bec112ad9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:39:18.304777 8429 raft_consensus.cc:482] T 22b87493dd324d2f947a14bfba1074d2 P d584780e230c4de9b74928bec112ad9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:39:18.304819 8429 raft_consensus.cc:3037] T 22b87493dd324d2f947a14bfba1074d2 P d584780e230c4de9b74928bec112ad9a [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:39:18.305935 8429 raft_consensus.cc:504] T 22b87493dd324d2f947a14bfba1074d2 P d584780e230c4de9b74928bec112ad9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } } peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } }
I20260927 02:39:18.306232 8429 leader_election.cc:288] T 22b87493dd324d2f947a14bfba1074d2 P d584780e230c4de9b74928bec112ad9a [CANDIDATE]: Term 1 election: Requested vote from peers 6feca25dab664c3ebcd5e224abf3e8a7 (127.0.98.2:39041), 97e535c7163e495a87620630469ab388 (127.0.98.3:41159)
I20260927 02:39:18.306581 8347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b87493dd324d2f947a14bfba1074d2" candidate_uuid: "d584780e230c4de9b74928bec112ad9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97e535c7163e495a87620630469ab388"
I20260927 02:39:18.306608 8207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b87493dd324d2f947a14bfba1074d2" candidate_uuid: "d584780e230c4de9b74928bec112ad9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6feca25dab664c3ebcd5e224abf3e8a7"
I20260927 02:39:18.306692 8347 raft_consensus.cc:3037] T 22b87493dd324d2f947a14bfba1074d2 P 97e535c7163e495a87620630469ab388 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:39:18.306717 8207 raft_consensus.cc:3037] T 22b87493dd324d2f947a14bfba1074d2 P 6feca25dab664c3ebcd5e224abf3e8a7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:39:18.307520 8347 raft_consensus.cc:2454] T 22b87493dd324d2f947a14bfba1074d2 P 97e535c7163e495a87620630469ab388 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d584780e230c4de9b74928bec112ad9a in term 1.
I20260927 02:39:18.307629 8207 raft_consensus.cc:2454] T 22b87493dd324d2f947a14bfba1074d2 P 6feca25dab664c3ebcd5e224abf3e8a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d584780e230c4de9b74928bec112ad9a in term 1.
I20260927 02:39:18.307754 7998 leader_election.cc:302] T 22b87493dd324d2f947a14bfba1074d2 P d584780e230c4de9b74928bec112ad9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97e535c7163e495a87620630469ab388, d584780e230c4de9b74928bec112ad9a; no voters:
I20260927 02:39:18.307961 8429 raft_consensus.cc:2781] T 22b87493dd324d2f947a14bfba1074d2 P d584780e230c4de9b74928bec112ad9a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:39:18.308133 8429 raft_consensus.cc:686] T 22b87493dd324d2f947a14bfba1074d2 P d584780e230c4de9b74928bec112ad9a [term 1 LEADER]: Becoming Leader. State: Replica: d584780e230c4de9b74928bec112ad9a, State: Running, Role: LEADER
I20260927 02:39:18.308368 8429 consensus_queue.cc:237] T 22b87493dd324d2f947a14bfba1074d2 P d584780e230c4de9b74928bec112ad9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } } peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } }
I20260927 02:39:18.310540 7910 catalog_manager.cc:5495] T 22b87493dd324d2f947a14bfba1074d2 P d584780e230c4de9b74928bec112ad9a reported cstate change: term changed from 0 to 1, leader changed from <none> to d584780e230c4de9b74928bec112ad9a (127.0.98.1). New cstate: current_term: 1 leader_uuid: "d584780e230c4de9b74928bec112ad9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d584780e230c4de9b74928bec112ad9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 } health_report { overall_health: UNKNOWN } } }
I20260927 02:39:18.330571 8520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:18.326525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49640 (local address 127.0.98.1:44417)
0927 02:39:18.326969 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:18.326975 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:18.326985 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:39:18.327069 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:18.327071 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:18.327120 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:18.327215 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:39:18.327440 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:18.327976 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:18.328849 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:18.329187 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:18.329271 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:18.329498 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:39:18.330118 (+ 620us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:39:18.330282 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:18.330346 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 02:39:18.330397 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":94,"threads_started":1}
I20260927 02:39:18.340888 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:18.337662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60509 (local address 127.0.98.2:39041)
0927 02:39:18.337765 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:18.337768 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:18.337897 (+ 129us) server_negotiation.cc:407] Connection header received
0927 02:39:18.338004 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:18.338006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:18.338043 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:18.338115 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:39:18.338325 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:18.338873 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:18.339750 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:18.340075 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:18.340187 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:18.340494 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:39:18.340573 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:39:18.340649 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:18.340741 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 02:39:18.340783 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 02:39:18.340916 8523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:18.337581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39041 (local address 127.0.98.1:60509)
0927 02:39:18.337842 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:18.337855 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:18.337934 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:18.338125 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:18.338127 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:18.338134 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:39:18.338230 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:18.338234 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:18.338983 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:18.338986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:18.339635 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:18.339647 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:18.340209 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:18.340213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:18.340307 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:18.340359 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:39:18.340659 (+ 300us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:39:18.340664 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:39:18.340731 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 02:39:18.340795 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":77,"threads_started":1}
I20260927 02:39:18.346294 8347 raft_consensus.cc:1261] T 22b87493dd324d2f947a14bfba1074d2 P 97e535c7163e495a87620630469ab388 [term 1 FOLLOWER]: Refusing update from remote peer d584780e230c4de9b74928bec112ad9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:39:18.346359 8207 raft_consensus.cc:1261] T 22b87493dd324d2f947a14bfba1074d2 P 6feca25dab664c3ebcd5e224abf3e8a7 [term 1 FOLLOWER]: Refusing update from remote peer d584780e230c4de9b74928bec112ad9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:39:18.347009 8518 consensus_queue.cc:1035] T 22b87493dd324d2f947a14bfba1074d2 P d584780e230c4de9b74928bec112ad9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6feca25dab664c3ebcd5e224abf3e8a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:39:18.347195 8429 consensus_queue.cc:1035] T 22b87493dd324d2f947a14bfba1074d2 P d584780e230c4de9b74928bec112ad9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "97e535c7163e495a87620630469ab388" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:39:18.363677 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:18.360410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44417 (local address 127.0.98.2:42013)
0927 02:39:18.360537 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:18.360553 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:18.360637 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:18.360842 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:18.360845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:18.360851 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:39:18.360935 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:18.360940 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:18.361695 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:18.361698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:18.362428 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:18.362438 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:18.363011 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:18.363015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:18.363097 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:18.363129 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:39:18.363401 (+ 272us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:39:18.363408 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 02:39:18.363468 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 02:39:18.363540 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 02:39:18.363718 8520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:18.360513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42013 (local address 127.0.98.1:44417)
0927 02:39:18.360661 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:18.360664 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:18.360677 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:39:18.360713 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:18.360714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:18.360756 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:18.360828 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:39:18.361031 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:18.361604 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:18.362541 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:18.362886 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:18.362983 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:18.363244 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:39:18.363324 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:39:18.363378 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:18.363495 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 02:39:18.363544 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 02:39:18.380369 7909 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:39:18.380569 7909 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:39:18.382650 7909 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c03c951086c4af5b892870108fd8990: Sending DeleteTablet for 3 replicas of tablet 22b87493dd324d2f947a14bfba1074d2
I20260927 02:39:18.383178 8047 tablet_service.cc:1514] Processing DeleteTablet for tablet 22b87493dd324d2f947a14bfba1074d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:39:18 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49626
I20260927 02:39:18.383347 8491 tablet_replica.cc:330] T 22b87493dd324d2f947a14bfba1074d2 P d584780e230c4de9b74928bec112ad9a: stopping tablet replica
I20260927 02:39:18.383327 8327 tablet_service.cc:1514] Processing DeleteTablet for tablet 22b87493dd324d2f947a14bfba1074d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:39:18 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54358
I20260927 02:39:18.383455 8491 raft_consensus.cc:2229] T 22b87493dd324d2f947a14bfba1074d2 P d584780e230c4de9b74928bec112ad9a [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:39:18.383495 8493 tablet_replica.cc:330] T 22b87493dd324d2f947a14bfba1074d2 P 97e535c7163e495a87620630469ab388: stopping tablet replica
I20260927 02:39:18.383594 8493 raft_consensus.cc:2229] T 22b87493dd324d2f947a14bfba1074d2 P 97e535c7163e495a87620630469ab388 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:39:18.383704 8493 raft_consensus.cc:2258] T 22b87493dd324d2f947a14bfba1074d2 P 97e535c7163e495a87620630469ab388 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:39:18.383718 8491 raft_consensus.cc:2258] T 22b87493dd324d2f947a14bfba1074d2 P d584780e230c4de9b74928bec112ad9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:39:18.383975 8187 tablet_service.cc:1514] Processing DeleteTablet for tablet 22b87493dd324d2f947a14bfba1074d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:39:18 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45594
I20260927 02:39:18.384143 8492 tablet_replica.cc:330] T 22b87493dd324d2f947a14bfba1074d2 P 6feca25dab664c3ebcd5e224abf3e8a7: stopping tablet replica
I20260927 02:39:18.384192 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 7982
I20260927 02:39:18.384243 8492 raft_consensus.cc:2229] T 22b87493dd324d2f947a14bfba1074d2 P 6feca25dab664c3ebcd5e224abf3e8a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:39:18.384349 8492 raft_consensus.cc:2258] T 22b87493dd324d2f947a14bfba1074d2 P 6feca25dab664c3ebcd5e224abf3e8a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:39:18.386021 8492 ts_tablet_manager.cc:1905] T 22b87493dd324d2f947a14bfba1074d2 P 6feca25dab664c3ebcd5e224abf3e8a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:39:18.388638 8492 ts_tablet_manager.cc:1918] T 22b87493dd324d2f947a14bfba1074d2 P 6feca25dab664c3ebcd5e224abf3e8a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:39:18.388722 8492 log.cc:1196] T 22b87493dd324d2f947a14bfba1074d2 P 6feca25dab664c3ebcd5e224abf3e8a7: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-1/wal/wals/22b87493dd324d2f947a14bfba1074d2
I20260927 02:39:18.388939 8492 ts_tablet_manager.cc:1939] T 22b87493dd324d2f947a14bfba1074d2 P 6feca25dab664c3ebcd5e224abf3e8a7: Deleting consensus metadata
I20260927 02:39:18.389199 8493 ts_tablet_manager.cc:1905] T 22b87493dd324d2f947a14bfba1074d2 P 97e535c7163e495a87620630469ab388: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:39:18.389859 7897 catalog_manager.cc:4841] TS 6feca25dab664c3ebcd5e224abf3e8a7 (127.0.98.2:39041): tablet 22b87493dd324d2f947a14bfba1074d2 (table test-table [id=d53440acec204333948422006c747ae1]) successfully deleted
I20260927 02:39:18.391243 8493 ts_tablet_manager.cc:1918] T 22b87493dd324d2f947a14bfba1074d2 P 97e535c7163e495a87620630469ab388: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:39:18.391324 8493 log.cc:1196] T 22b87493dd324d2f947a14bfba1074d2 P 97e535c7163e495a87620630469ab388: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790476718023131-392-0/minicluster-data/ts-2/wal/wals/22b87493dd324d2f947a14bfba1074d2
I20260927 02:39:18.391578 8493 ts_tablet_manager.cc:1939] T 22b87493dd324d2f947a14bfba1074d2 P 97e535c7163e495a87620630469ab388: Deleting consensus metadata
W20260927 02:39:18.391727 7895 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44417 (error 108)
I20260927 02:39:18.392271 7894 catalog_manager.cc:4841] TS 97e535c7163e495a87620630469ab388 (127.0.98.3:41159): tablet 22b87493dd324d2f947a14bfba1074d2 (table test-table [id=d53440acec204333948422006c747ae1]) successfully deleted
I20260927 02:39:18.392511 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 8122
I20260927 02:39:18.392709 8529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:18.392369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44417 (local address 127.0.0.1:49650)
0927 02:39:18.392549 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:18.392617 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":58,"threads_started":1}
W20260927 02:39:18.393918 7895 catalog_manager.cc:4584] TS d584780e230c4de9b74928bec112ad9a (127.0.98.1:44417): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 22b87493dd324d2f947a14bfba1074d2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44417: connect: Connection refused (error 111)
I20260927 02:39:18.398769 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 8262
I20260927 02:39:18.404202 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 7878
2026-09-27T02:39:18Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3683 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476718023131-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:39:18 dist-test-slave-zv8z krb5kdc[8532](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:39:18 dist-test-slave-zv8z krb5kdc[8532](info): set up 2 sockets
Sep 27 02:39:18 dist-test-slave-zv8z krb5kdc[8532](info): commencing operation
krb5kdc: starting...
W20260927 02:39:19.366624 392 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476718023131-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476718023131-392-0/krb5kdc/test-user.keytab.
Sep 27 02:39:19 dist-test-slave-zv8z krb5kdc[8532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:39:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:39:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:39:19.479406 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45137
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:39:19.585449 8545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:39:19.585741 8545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:39:19.585803 8545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:39:19.589800 8545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:39:19.589860 8545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:39:19.589886 8545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:39:19.589907 8545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:39:19.589927 8545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:39:19.595146 8545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:39:19.596148 8545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:39:19.597138 8545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:39:19.605263 8551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:19.605280 8545 server_base.cc:1034] running on GCE node
W20260927 02:39:19.605297 8550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:19.605332 8553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:19.606182 8545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:39:19.607278 8545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:39:19.608450 8545 hybrid_clock.cc:648] HybridClock initialized: now 1790476759608436 us; error 28 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T02:39:19Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1212 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:39:19 dist-test-slave-zv8z krb5kdc[8558](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:39:19 dist-test-slave-zv8z krb5kdc[8558](info): set up 2 sockets
Sep 27 02:39:19 dist-test-slave-zv8z krb5kdc[8558](info): commencing operation
krb5kdc: starting...
W20260927 02:39:20.566617 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/test-user.keytab.
Sep 27 02:39:20 dist-test-slave-zv8z krb5kdc[8558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:39:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:39:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:39:20.680769 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42451
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42451
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:39:20.786893 8571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:39:20.787127 8571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:39:20.787163 8571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:39:20.790998 8571 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:39:20.791041 8571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:39:20.791059 8571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:39:20.791074 8571 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:39:20.791086 8571 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:39:20.796093 8571 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:39:20.797017 8571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:39:20.797940 8571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:39:20.806124 8577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:20.806188 8571 server_base.cc:1034] running on GCE node
W20260927 02:39:20.806157 8576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:20.806124 8579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:20.806921 8571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:39:20.807978 8571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:39:20.809136 8571 hybrid_clock.cc:648] HybridClock initialized: now 1790476760809123 us; error 27 us; skew 500 ppm
Sep 27 02:39:20 dist-test-slave-zv8z krb5kdc[8558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476760, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:39:20.811764 8571 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:39:20.812608 8571 webserver.cc:460] Webserver started at http://127.0.98.62:40649/ using document root <none> and password file <none>
I20260927 02:39:20.813117 8571 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:20.813180 8571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:20.813374 8571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:39:20.815339 8571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/master-0/data/instance:
uuid: "03ac27e88d1e4cbbb9bbeba7cacaca72"
format_stamp: "Formatted at 2026-09-27 02:39:20 on dist-test-slave-zv8z"
server_key: "5e72d0c522a10295a2691c8294cb31ba"
server_key_iv: "df3e0e4ab1fd89e81ccc5cde0507ee18"
server_key_version: "encryptionkey@0"
I20260927 02:39:20.815850 8571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/master-0/wal/instance:
uuid: "03ac27e88d1e4cbbb9bbeba7cacaca72"
format_stamp: "Formatted at 2026-09-27 02:39:20 on dist-test-slave-zv8z"
server_key: "5e72d0c522a10295a2691c8294cb31ba"
server_key_iv: "df3e0e4ab1fd89e81ccc5cde0507ee18"
server_key_version: "encryptionkey@0"
I20260927 02:39:20.819285 8571 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:39:20.821508 8586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:20.822688 8571 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:39:20.822794 8571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "03ac27e88d1e4cbbb9bbeba7cacaca72"
format_stamp: "Formatted at 2026-09-27 02:39:20 on dist-test-slave-zv8z"
server_key: "5e72d0c522a10295a2691c8294cb31ba"
server_key_iv: "df3e0e4ab1fd89e81ccc5cde0507ee18"
server_key_version: "encryptionkey@0"
I20260927 02:39:20.822885 8571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:20.857017 8571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:39:20.860072 8571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:39:20.860240 8571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:39:20.866597 8571 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42451
I20260927 02:39:20.866626 8638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42451 every 8 connection(s)
I20260927 02:39:20.867476 8571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:39:20.870019 8639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:39:20.875286 8639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03ac27e88d1e4cbbb9bbeba7cacaca72: Bootstrap starting.
I20260927 02:39:20.876713 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 8571
I20260927 02:39:20.876832 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/master-0/wal/instance
I20260927 02:39:20.877121 392 external_mini_cluster.cc:1428] Setting key 7458faef088b28bf884336a8bee11b90
I20260927 02:39:20.877910 8639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03ac27e88d1e4cbbb9bbeba7cacaca72: Neither blocks nor log segments found. Creating new log.
I20260927 02:39:20.878764 8639 log.cc:824] T 00000000000000000000000000000000 P 03ac27e88d1e4cbbb9bbeba7cacaca72: Log is configured to *not* fsync() on all Append() calls
I20260927 02:39:20.880627 8639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03ac27e88d1e4cbbb9bbeba7cacaca72: No bootstrap required, opened a new log
I20260927 02:39:20.883911 8639 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03ac27e88d1e4cbbb9bbeba7cacaca72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03ac27e88d1e4cbbb9bbeba7cacaca72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42451 } }
I20260927 02:39:20.884112 8639 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03ac27e88d1e4cbbb9bbeba7cacaca72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:39:20.884145 8639 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03ac27e88d1e4cbbb9bbeba7cacaca72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03ac27e88d1e4cbbb9bbeba7cacaca72, State: Initialized, Role: FOLLOWER
I20260927 02:39:20.884558 8639 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03ac27e88d1e4cbbb9bbeba7cacaca72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03ac27e88d1e4cbbb9bbeba7cacaca72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42451 } }
I20260927 02:39:20.884677 8639 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03ac27e88d1e4cbbb9bbeba7cacaca72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:39:20.884735 8639 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03ac27e88d1e4cbbb9bbeba7cacaca72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:39:20.884824 8639 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03ac27e88d1e4cbbb9bbeba7cacaca72 [term 0 FOLLOWER]: Advancing to term 1
Sep 27 02:39:20 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:39:20.885643 8639 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03ac27e88d1e4cbbb9bbeba7cacaca72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03ac27e88d1e4cbbb9bbeba7cacaca72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42451 } }
I20260927 02:39:20.885946 8639 leader_election.cc:302] T 00000000000000000000000000000000 P 03ac27e88d1e4cbbb9bbeba7cacaca72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03ac27e88d1e4cbbb9bbeba7cacaca72; no voters:
I20260927 02:39:20.886206 8639 leader_election.cc:288] T 00000000000000000000000000000000 P 03ac27e88d1e4cbbb9bbeba7cacaca72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:39:20.886354 8644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03ac27e88d1e4cbbb9bbeba7cacaca72 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:39:20.886533 8644 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03ac27e88d1e4cbbb9bbeba7cacaca72 [term 1 LEADER]: Becoming Leader. State: Replica: 03ac27e88d1e4cbbb9bbeba7cacaca72, State: Running, Role: LEADER
I20260927 02:39:20.886825 8644 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03ac27e88d1e4cbbb9bbeba7cacaca72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03ac27e88d1e4cbbb9bbeba7cacaca72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42451 } }
I20260927 02:39:20.887455 8639 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03ac27e88d1e4cbbb9bbeba7cacaca72 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:39:20.888411 8646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03ac27e88d1e4cbbb9bbeba7cacaca72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03ac27e88d1e4cbbb9bbeba7cacaca72. Latest consensus state: current_term: 1 leader_uuid: "03ac27e88d1e4cbbb9bbeba7cacaca72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03ac27e88d1e4cbbb9bbeba7cacaca72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42451 } } }
I20260927 02:39:20.888530 8646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03ac27e88d1e4cbbb9bbeba7cacaca72 [sys.catalog]: This master's current role is: LEADER
I20260927 02:39:20.888792 8645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03ac27e88d1e4cbbb9bbeba7cacaca72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03ac27e88d1e4cbbb9bbeba7cacaca72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03ac27e88d1e4cbbb9bbeba7cacaca72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42451 } } }
I20260927 02:39:20.888911 8645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03ac27e88d1e4cbbb9bbeba7cacaca72 [sys.catalog]: This master's current role is: LEADER
I20260927 02:39:20.889701 8653 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:39:20.891283 8642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:20.878242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38030 (local address 127.0.98.62:42451)
0927 02:39:20.878700 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:20.878714 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:20.878750 (+ 36us) server_negotiation.cc:407] Connection header received
0927 02:39:20.879574 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:20.879594 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:20.879934 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:20.880254 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:20.881014 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:20.881976 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:20.882833 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:20.883275 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:20.883424 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:20.885722 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:20.885726 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:20.885738 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:20.885765 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:20.887634 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:20.888119 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:20.888125 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:20.888131 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:20.888264 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:20.888670 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:20.888673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:20.888675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:20.889173 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:20.889367 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:20.890151 (+ 784us) server_negotiation.cc:299] Negotiation successful
0927 02:39:20.890460 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":126,"threads_started":1}
I20260927 02:39:20.892410 8653 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:39:20.897646 8653 catalog_manager.cc:1347] Generated new cluster ID: 2ed16bbdb83e4d21a429c1988ee57bd5
I20260927 02:39:20.897707 8653 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:39:20.909996 8653 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:39:20.911227 8653 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:39:20.927027 8653 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03ac27e88d1e4cbbb9bbeba7cacaca72: Generated new TSK 0
I20260927 02:39:20.927835 8653 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:39:20.957329 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42451
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:39:21.062849 8665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:39:21.063097 8665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:39:21.063138 8665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:39:21.067081 8665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:39:21.067131 8665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:39:21.067189 8665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:39:21.072202 8665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:39:21.073153 8665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:39:21.074105 8665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:39:21.083240 8670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:21.083305 8665 server_base.cc:1034] running on GCE node
W20260927 02:39:21.083248 8671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:21.083248 8673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:21.083890 8665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:39:21.084496 8665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:39:21.085678 8665 hybrid_clock.cc:648] HybridClock initialized: now 1790476761085658 us; error 31 us; skew 500 ppm
Sep 27 02:39:21 dist-test-slave-zv8z krb5kdc[8558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:39:21.088569 8665 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:39:21.089524 8665 webserver.cc:460] Webserver started at http://127.0.98.1:34657/ using document root <none> and password file <none>
I20260927 02:39:21.090081 8665 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:21.090145 8665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:21.090389 8665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:39:21.092533 8665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-0/data/instance:
uuid: "7be6a87ea13044d1876e74a992f2d47f"
format_stamp: "Formatted at 2026-09-27 02:39:21 on dist-test-slave-zv8z"
server_key: "7f1a46767baa17c666a4d316e9bc14c4"
server_key_iv: "bfe54c0ab9c93a28fa57745d2378a580"
server_key_version: "encryptionkey@0"
I20260927 02:39:21.093067 8665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7be6a87ea13044d1876e74a992f2d47f"
format_stamp: "Formatted at 2026-09-27 02:39:21 on dist-test-slave-zv8z"
server_key: "7f1a46767baa17c666a4d316e9bc14c4"
server_key_iv: "bfe54c0ab9c93a28fa57745d2378a580"
server_key_version: "encryptionkey@0"
I20260927 02:39:21.096642 8665 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:39:21.099094 8680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:21.100243 8665 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:39:21.100345 8665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-0/wal
uuid: "7be6a87ea13044d1876e74a992f2d47f"
format_stamp: "Formatted at 2026-09-27 02:39:21 on dist-test-slave-zv8z"
server_key: "7f1a46767baa17c666a4d316e9bc14c4"
server_key_iv: "bfe54c0ab9c93a28fa57745d2378a580"
server_key_version: "encryptionkey@0"
I20260927 02:39:21.100442 8665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:21.112891 8665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:39:21.115825 8665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:39:21.115993 8665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:39:21.116521 8665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:39:21.117285 8665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:39:21.117329 8665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:21.117377 8665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:39:21.117404 8665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:21.125473 8665 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33073
I20260927 02:39:21.125542 8793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33073 every 8 connection(s)
I20260927 02:39:21.126377 8665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:39:21.132843 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 8665
I20260927 02:39:21.132964 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:39:21.133279 392 external_mini_cluster.cc:1428] Setting key 55306c5c51803dec4c8ef93cc3963eee
Sep 27 02:39:21 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:39:21.140425 8642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:21.128072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60243 (local address 127.0.98.62:42451)
0927 02:39:21.128212 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:21.128217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:21.129239 (+ 1022us) server_negotiation.cc:407] Connection header received
0927 02:39:21.130586 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:21.130589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:21.130635 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:21.130705 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:21.131862 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:21.132394 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:21.133881 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:21.134158 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:21.134283 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:21.136908 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:21.136911 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:21.136913 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:21.136938 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:21.138367 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:21.138929 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:21.138932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:21.138933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:21.139000 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:21.139340 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:21.139341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:21.139342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:21.139523 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:21.139592 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:21.140206 (+ 614us) server_negotiation.cc:299] Negotiation successful
0927 02:39:21.140306 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 02:39:21.141592 8796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:21.128566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.1:60243)
0927 02:39:21.129058 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:21.129114 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:21.130393 (+ 1279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:21.130855 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:21.130861 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:21.131226 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:21.131713 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:21.131728 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:21.132528 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:21.132531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:21.133781 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:21.133790 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:21.134320 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:21.134324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:21.134488 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:21.135132 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:21.135149 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:21.136703 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:21.138468 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:21.138474 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:21.138491 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:21.138841 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:21.139074 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:21.139077 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:21.139079 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:21.139262 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:21.139614 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:21.139620 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:21.139946 (+ 326us) client_negotiation.cc:769] Sending connection context
0927 02:39:21.140162 (+ 216us) client_negotiation.cc:240] Negotiation successful
0927 02:39:21.140523 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":117,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:39:21.142817 8794 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42451
I20260927 02:39:21.143064 8794 heartbeater.cc:464] Registering TS with master...
I20260927 02:39:21.143617 8794 heartbeater.cc:511] Master 127.0.98.62:42451 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 02:39:21.145267 8603 ts_manager.cc:194] Registered new tserver with Master: 7be6a87ea13044d1876e74a992f2d47f (127.0.98.1:33073)
I20260927 02:39:21.146409 8603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60243
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:39:21.159196 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42451
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:39:21.267032 8799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:39:21.267282 8799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:39:21.267323 8799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:39:21.271214 8799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:39:21.271260 8799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:39:21.271311 8799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:39:21.276386 8799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:39:21.277333 8799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:39:21.278373 8799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:39:21.287554 8804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:21.287556 8805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:21.287647 8799 server_base.cc:1034] running on GCE node
W20260927 02:39:21.287583 8807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:21.288215 8799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:39:21.288921 8799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:39:21.290120 8799 hybrid_clock.cc:648] HybridClock initialized: now 1790476761290088 us; error 43 us; skew 500 ppm
Sep 27 02:39:21 dist-test-slave-zv8z krb5kdc[8558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:39:21.292843 8799 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:39:21.293722 8799 webserver.cc:460] Webserver started at http://127.0.98.2:35291/ using document root <none> and password file <none>
I20260927 02:39:21.294303 8799 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:21.294369 8799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:21.294570 8799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:39:21.296522 8799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-1/data/instance:
uuid: "68a6173a40704018b49cd71d19cc6d4a"
format_stamp: "Formatted at 2026-09-27 02:39:21 on dist-test-slave-zv8z"
server_key: "474a40247881f149f6c8a32bc4c73d3b"
server_key_iv: "1c6b7143607754fd7d55e1ed70aa983d"
server_key_version: "encryptionkey@0"
I20260927 02:39:21.297053 8799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-1/wal/instance:
uuid: "68a6173a40704018b49cd71d19cc6d4a"
format_stamp: "Formatted at 2026-09-27 02:39:21 on dist-test-slave-zv8z"
server_key: "474a40247881f149f6c8a32bc4c73d3b"
server_key_iv: "1c6b7143607754fd7d55e1ed70aa983d"
server_key_version: "encryptionkey@0"
I20260927 02:39:21.300621 8799 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 02:39:21.303197 8814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:21.304425 8799 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:39:21.304522 8799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-1/wal
uuid: "68a6173a40704018b49cd71d19cc6d4a"
format_stamp: "Formatted at 2026-09-27 02:39:21 on dist-test-slave-zv8z"
server_key: "474a40247881f149f6c8a32bc4c73d3b"
server_key_iv: "1c6b7143607754fd7d55e1ed70aa983d"
server_key_version: "encryptionkey@0"
I20260927 02:39:21.304612 8799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:21.320756 8799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:39:21.323891 8799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:39:21.324081 8799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:39:21.324649 8799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:39:21.325496 8799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:39:21.325541 8799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:21.325582 8799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:39:21.325609 8799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:21.333642 8799 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39351
I20260927 02:39:21.333666 8927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39351 every 8 connection(s)
I20260927 02:39:21.334535 8799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:39:21.334865 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 8799
I20260927 02:39:21.334980 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:39:21.335285 392 external_mini_cluster.cc:1428] Setting key 6d606a0e52abdb63dce28901eeed1711
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Sep 27 02:39:21 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 02:39:21.348475 8642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:21.336330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53815 (local address 127.0.98.62:42451)
0927 02:39:21.336435 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:21.336438 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:21.337483 (+ 1045us) server_negotiation.cc:407] Connection header received
0927 02:39:21.339134 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:21.339136 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:21.339181 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:21.339240 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:21.340687 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:21.341245 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:21.342441 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:21.342660 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:21.342751 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:21.345078 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:21.345081 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:21.345083 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:21.345109 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:21.346484 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:21.347052 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:21.347056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:21.347057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:21.347132 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:21.347429 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:21.347433 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:21.347435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:21.347606 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:21.347689 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:21.348250 (+ 561us) server_negotiation.cc:299] Negotiation successful
0927 02:39:21.348355 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 02:39:21.349521 8930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:21.336870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.2:53815)
0927 02:39:21.337315 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:21.337374 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:21.338897 (+ 1523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:21.339445 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:21.339453 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:21.339895 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:21.340527 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:21.340548 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:21.341385 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:21.341389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:21.342324 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:21.342334 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:21.342794 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:21.342798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:21.342968 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:21.343492 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:21.343508 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:21.344880 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:21.346621 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:21.346626 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:21.346638 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:21.346955 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:21.347211 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:21.347214 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:21.347215 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:21.347343 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:21.347700 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:21.347707 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:21.347992 (+ 285us) client_negotiation.cc:769] Sending connection context
0927 02:39:21.348207 (+ 215us) client_negotiation.cc:240] Negotiation successful
0927 02:39:21.348553 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":107,"threads_started":1}
I20260927 02:39:21.350714 8928 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42451
I20260927 02:39:21.351002 8928 heartbeater.cc:464] Registering TS with master...
I20260927 02:39:21.351493 8928 heartbeater.cc:511] Master 127.0.98.62:42451 requested a full tablet report, sending...
I20260927 02:39:21.352448 8603 ts_manager.cc:194] Registered new tserver with Master: 68a6173a40704018b49cd71d19cc6d4a (127.0.98.2:39351)
I20260927 02:39:21.352994 8603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53815
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:39:21.361622 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42451
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:39:21.468569 8933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:39:21.468816 8933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:39:21.468856 8933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:39:21.472769 8933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:39:21.472817 8933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:39:21.472872 8933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:39:21.477869 8933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:39:21.478819 8933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:39:21.479741 8933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:39:21.488848 8938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:21.488883 8939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:21.488893 8941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:21.489142 8933 server_base.cc:1034] running on GCE node
I20260927 02:39:21.489559 8933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:39:21.490216 8933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:39:21.491393 8933 hybrid_clock.cc:648] HybridClock initialized: now 1790476761491380 us; error 35 us; skew 500 ppm
Sep 27 02:39:21 dist-test-slave-zv8z krb5kdc[8558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:39:21.494103 8933 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:39:21.494997 8933 webserver.cc:460] Webserver started at http://127.0.98.3:37341/ using document root <none> and password file <none>
I20260927 02:39:21.495522 8933 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:21.495589 8933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:21.495766 8933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:39:21.497737 8933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-2/data/instance:
uuid: "14870d4394664dfa951acda7dc1680d2"
format_stamp: "Formatted at 2026-09-27 02:39:21 on dist-test-slave-zv8z"
server_key: "8fe2be6b1ea8fc34923a9e991192c2cf"
server_key_iv: "02902c03e2ce08078800972e996fb3da"
server_key_version: "encryptionkey@0"
I20260927 02:39:21.498281 8933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-2/wal/instance:
uuid: "14870d4394664dfa951acda7dc1680d2"
format_stamp: "Formatted at 2026-09-27 02:39:21 on dist-test-slave-zv8z"
server_key: "8fe2be6b1ea8fc34923a9e991192c2cf"
server_key_iv: "02902c03e2ce08078800972e996fb3da"
server_key_version: "encryptionkey@0"
I20260927 02:39:21.502050 8933 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260927 02:39:21.504510 8948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:21.505743 8933 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260927 02:39:21.505856 8933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-2/wal
uuid: "14870d4394664dfa951acda7dc1680d2"
format_stamp: "Formatted at 2026-09-27 02:39:21 on dist-test-slave-zv8z"
server_key: "8fe2be6b1ea8fc34923a9e991192c2cf"
server_key_iv: "02902c03e2ce08078800972e996fb3da"
server_key_version: "encryptionkey@0"
I20260927 02:39:21.505947 8933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:21.529709 8933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:39:21.532778 8933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:39:21.532948 8933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:39:21.533507 8933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:39:21.534415 8933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:39:21.534478 8933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:21.534534 8933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:39:21.534557 8933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:21.543368 8933 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33111
I20260927 02:39:21.543437 9061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33111 every 8 connection(s)
I20260927 02:39:21.544241 8933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:39:21.547477 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 8933
I20260927 02:39:21.547606 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:39:21.547904 392 external_mini_cluster.cc:1428] Setting key a5c894413482d61eb810b4b33bb8e8e5
Sep 27 02:39:21 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:39:21.557480 8642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:21.545998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58907 (local address 127.0.98.62:42451)
0927 02:39:21.546172 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:21.546176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:21.547051 (+ 875us) server_negotiation.cc:407] Connection header received
0927 02:39:21.548614 (+ 1563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:21.548617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:21.548661 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:21.548743 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:21.550173 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:21.550747 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:21.551848 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:21.552059 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:21.552133 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:21.554450 (+ 2317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:21.554453 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:21.554455 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:21.554478 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:21.555773 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:21.556243 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:21.556246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:21.556247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:21.556309 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:21.556569 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:21.556571 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:21.556572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:21.556730 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:21.556800 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:21.557265 (+ 465us) server_negotiation.cc:299] Negotiation successful
0927 02:39:21.557363 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 02:39:21.558295 9064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:21.546416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.3:58907)
0927 02:39:21.546868 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:21.546909 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:21.548393 (+ 1484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:21.548912 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:21.548919 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:21.549370 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:21.550030 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:21.550050 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:21.550859 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:21.550862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:21.551756 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:21.551765 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:21.552205 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:21.552208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:21.552349 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:21.552871 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:21.552887 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:21.554288 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:21.555886 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:21.555890 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:21.555902 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:21.556168 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:21.556372 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:21.556373 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:21.556375 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:21.556503 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:21.556804 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:21.556809 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:21.557040 (+ 231us) client_negotiation.cc:769] Sending connection context
0927 02:39:21.557189 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 02:39:21.557447 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":126,"threads_started":1}
I20260927 02:39:21.559439 9062 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42451
I20260927 02:39:21.559665 9062 heartbeater.cc:464] Registering TS with master...
I20260927 02:39:21.560119 9062 heartbeater.cc:511] Master 127.0.98.62:42451 requested a full tablet report, sending...
I20260927 02:39:21.561034 8603 ts_manager.cc:194] Registered new tserver with Master: 14870d4394664dfa951acda7dc1680d2 (127.0.98.3:33111)
I20260927 02:39:21.561650 8603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58907
I20260927 02:39:21.561672 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 02:39:21.628690 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 8665
I20260927 02:39:21.633872 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 8799
I20260927 02:39:21.638209 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 8933
I20260927 02:39:21.642620 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 8571
I20260927 02:39:21.647583 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42451
--webserver_interface=127.0.98.62
--webserver_port=40649
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42451
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:39:21.755945 9071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:39:21.756189 9071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:39:21.756227 9071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:39:21.760190 9071 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:39:21.760242 9071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:39:21.760259 9071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:39:21.760274 9071 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:39:21.760289 9071 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:39:21.765494 9071 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=40649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:39:21.766515 9071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:39:21.767477 9071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:39:21.775434 9077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:21.775431 9076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:21.775627 9071 server_base.cc:1034] running on GCE node
W20260927 02:39:21.775453 9079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:21.776332 9071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:39:21.777326 9071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:39:21.778469 9071 hybrid_clock.cc:648] HybridClock initialized: now 1790476761778444 us; error 40 us; skew 500 ppm
Sep 27 02:39:21 dist-test-slave-zv8z krb5kdc[8558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:39:21.781165 9071 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:39:21.781989 9071 webserver.cc:460] Webserver started at http://127.0.98.62:40649/ using document root <none> and password file <none>
I20260927 02:39:21.782582 9071 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:21.782661 9071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:21.786314 9071 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.003s
I20260927 02:39:21.788357 9086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:21.789537 9071 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:39:21.789649 9071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "03ac27e88d1e4cbbb9bbeba7cacaca72"
format_stamp: "Formatted at 2026-09-27 02:39:20 on dist-test-slave-zv8z"
server_key: "5e72d0c522a10295a2691c8294cb31ba"
server_key_iv: "df3e0e4ab1fd89e81ccc5cde0507ee18"
server_key_version: "encryptionkey@0"
I20260927 02:39:21.790149 9071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:21.816658 9071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:39:21.819494 9071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:39:21.819674 9071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:39:21.826354 9071 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42451
I20260927 02:39:21.826375 9138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42451 every 8 connection(s)
I20260927 02:39:21.827369 9071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:39:21.831413 9139 sys_catalog.cc:263] Verifying existing consensus state
I20260927 02:39:21.832947 9139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03ac27e88d1e4cbbb9bbeba7cacaca72: Bootstrap starting.
I20260927 02:39:21.833138 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 9071
I20260927 02:39:21.833520 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:33073
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42451
--webserver_port=34657
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:39:21.848809 9139 log.cc:824] T 00000000000000000000000000000000 P 03ac27e88d1e4cbbb9bbeba7cacaca72: Log is configured to *not* fsync() on all Append() calls
I20260927 02:39:21.855228 9139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03ac27e88d1e4cbbb9bbeba7cacaca72: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:39:21.855554 9139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03ac27e88d1e4cbbb9bbeba7cacaca72: Bootstrap complete.
I20260927 02:39:21.859541 9139 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03ac27e88d1e4cbbb9bbeba7cacaca72 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03ac27e88d1e4cbbb9bbeba7cacaca72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42451 } }
I20260927 02:39:21.859926 9139 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03ac27e88d1e4cbbb9bbeba7cacaca72 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03ac27e88d1e4cbbb9bbeba7cacaca72, State: Initialized, Role: FOLLOWER
I20260927 02:39:21.860359 9139 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03ac27e88d1e4cbbb9bbeba7cacaca72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03ac27e88d1e4cbbb9bbeba7cacaca72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42451 } }
I20260927 02:39:21.860471 9139 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03ac27e88d1e4cbbb9bbeba7cacaca72 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:39:21.860543 9139 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03ac27e88d1e4cbbb9bbeba7cacaca72 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:39:21.860630 9139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03ac27e88d1e4cbbb9bbeba7cacaca72 [term 1 FOLLOWER]: Advancing to term 2
I20260927 02:39:21.862457 9139 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03ac27e88d1e4cbbb9bbeba7cacaca72 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03ac27e88d1e4cbbb9bbeba7cacaca72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42451 } }
I20260927 02:39:21.862753 9139 leader_election.cc:302] T 00000000000000000000000000000000 P 03ac27e88d1e4cbbb9bbeba7cacaca72 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03ac27e88d1e4cbbb9bbeba7cacaca72; no voters:
I20260927 02:39:21.863068 9139 leader_election.cc:288] T 00000000000000000000000000000000 P 03ac27e88d1e4cbbb9bbeba7cacaca72 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 02:39:21.863171 9143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03ac27e88d1e4cbbb9bbeba7cacaca72 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 02:39:21.863353 9143 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03ac27e88d1e4cbbb9bbeba7cacaca72 [term 2 LEADER]: Becoming Leader. State: Replica: 03ac27e88d1e4cbbb9bbeba7cacaca72, State: Running, Role: LEADER
I20260927 02:39:21.863688 9143 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03ac27e88d1e4cbbb9bbeba7cacaca72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03ac27e88d1e4cbbb9bbeba7cacaca72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42451 } }
I20260927 02:39:21.864256 9139 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03ac27e88d1e4cbbb9bbeba7cacaca72 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:39:21.864552 9145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03ac27e88d1e4cbbb9bbeba7cacaca72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03ac27e88d1e4cbbb9bbeba7cacaca72. Latest consensus state: current_term: 2 leader_uuid: "03ac27e88d1e4cbbb9bbeba7cacaca72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03ac27e88d1e4cbbb9bbeba7cacaca72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42451 } } }
I20260927 02:39:21.864737 9145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03ac27e88d1e4cbbb9bbeba7cacaca72 [sys.catalog]: This master's current role is: LEADER
I20260927 02:39:21.864895 9144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03ac27e88d1e4cbbb9bbeba7cacaca72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "03ac27e88d1e4cbbb9bbeba7cacaca72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03ac27e88d1e4cbbb9bbeba7cacaca72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42451 } } }
I20260927 02:39:21.865006 9144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03ac27e88d1e4cbbb9bbeba7cacaca72 [sys.catalog]: This master's current role is: LEADER
I20260927 02:39:21.865236 9147 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:39:21.868338 9147 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:39:21.868960 9147 catalog_manager.cc:1259] Loaded cluster ID: 2ed16bbdb83e4d21a429c1988ee57bd5
I20260927 02:39:21.869009 9147 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:39:21.870671 9147 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:39:21.871191 9147 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 03ac27e88d1e4cbbb9bbeba7cacaca72: Loaded TSK: 0
I20260927 02:39:21.872057 9147 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 02:39:21.953097 9141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:39:21.953346 9141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:39:21.953405 9141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:39:21.957413 9141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:39:21.957475 9141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:39:21.957552 9141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:39:21.962709 9141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:33073
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34657
--tserver_master_addrs=127.0.98.62:42451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:39:21.963629 9141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:39:21.964648 9141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:39:21.974771 9165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:21.974784 9166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:21.974980 9141 server_base.cc:1034] running on GCE node
W20260927 02:39:21.975268 9168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:21.975677 9141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:39:21.976418 9141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:39:21.977620 9141 hybrid_clock.cc:648] HybridClock initialized: now 1790476761977589 us; error 38 us; skew 500 ppm
Sep 27 02:39:21 dist-test-slave-zv8z krb5kdc[8558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476761, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:39:21.980907 9141 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 02:39:21.981981 9141 webserver.cc:460] Webserver started at http://127.0.98.1:34657/ using document root <none> and password file <none>
I20260927 02:39:21.982686 9141 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:21.982791 9141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:21.987561 9141 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:39:21.989964 9175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:21.991446 9141 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:39:21.991576 9141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-0/wal
uuid: "7be6a87ea13044d1876e74a992f2d47f"
format_stamp: "Formatted at 2026-09-27 02:39:21 on dist-test-slave-zv8z"
server_key: "7f1a46767baa17c666a4d316e9bc14c4"
server_key_iv: "bfe54c0ab9c93a28fa57745d2378a580"
server_key_version: "encryptionkey@0"
I20260927 02:39:21.992259 9141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:22.023404 9141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:39:22.026916 9141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:39:22.027072 9141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:39:22.027611 9141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:39:22.028409 9141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:39:22.028455 9141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:22.028517 9141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:39:22.028566 9141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:22.036545 9141 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33073
I20260927 02:39:22.036589 9288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33073 every 8 connection(s)
I20260927 02:39:22.037495 9141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:39:22.039708 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 9141
I20260927 02:39:22.040175 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39351
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42451
--webserver_port=35291
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 02:39:22 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:22 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:39:22.051164 9292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:22.039674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.1:42067)
0927 02:39:22.040122 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:22.040161 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:22.041391 (+ 1230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:22.042458 (+ 1067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:22.042467 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:22.042865 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:22.043303 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:22.043317 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:22.045229 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:22.045232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:22.046081 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:22.046090 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:22.047166 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:22.047169 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:22.047314 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:22.047863 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:22.047879 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:22.050415 (+ 2536us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":113,"threads_started":1}
W20260927 02:39:22.051497 9291 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:22.039509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42067 (local address 127.0.98.62:42451)
0927 02:39:22.039900 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:22.039919 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:22.040270 (+ 351us) server_negotiation.cc:407] Connection header received
0927 02:39:22.041775 (+ 1505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:22.041786 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:22.042070 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:22.042391 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:22.043500 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:22.045078 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:22.046215 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:22.046987 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:22.047157 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:22.050669 (+ 3512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42067: BlockingRecv error: recv got EOF from 127.0.98.1:42067 (error 108)
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":91,"threads_started":1}
W20260927 02:39:22.051712 9289 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42451 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:42451: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 02:39:22 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:22 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:39:22.057410 9292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:22.052136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.1:39933)
0927 02:39:22.052306 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:22.052318 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:22.052378 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:22.052606 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:22.052609 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:22.052732 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:22.052820 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:22.052824 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:22.053802 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:22.053805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:22.054426 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:22.054435 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:22.054966 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:22.054969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:22.055031 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:22.055491 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:22.055501 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:22.057313 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106}
W20260927 02:39:22.057600 9291 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:22.052204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39933 (local address 127.0.98.62:42451)
0927 02:39:22.052343 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:22.052347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:22.052362 (+ 15us) server_negotiation.cc:407] Connection header received
0927 02:39:22.052454 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:22.052457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:22.052504 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:22.052589 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:22.052906 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:22.053699 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:22.054527 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:22.054882 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:22.054972 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:22.057474 (+ 2502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39933: BlockingRecv error: recv got EOF from 127.0.98.1:39933 (error 108)
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":59}
Sep 27 02:39:22 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:22 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:39:22.063071 9292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:22.058058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.1:47883)
0927 02:39:22.058209 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:22.058223 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:22.058324 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:22.058527 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:22.058529 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:22.058656 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:22.058724 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:22.058729 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:22.059663 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:22.059666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:22.060198 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:22.060205 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:22.060766 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:22.060769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:22.060825 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:22.061285 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:22.061296 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:22.062973 (+ 1677us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 02:39:22.063253 9289 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 02:39:22.063320 9291 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:22.058163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47883 (local address 127.0.98.62:42451)
0927 02:39:22.058301 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:22.058305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:22.058320 (+ 15us) server_negotiation.cc:407] Connection header received
0927 02:39:22.058388 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:22.058391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:22.058432 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:22.058501 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:22.058809 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:22.059554 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:22.060319 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:22.060666 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:22.060759 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:22.063119 (+ 2360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47883: BlockingRecv error: recv got EOF from 127.0.98.1:47883 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260927 02:39:22.150635 9293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:39:22.150882 9293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:39:22.150921 9293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:39:22.154840 9293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:39:22.154892 9293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:39:22.154946 9293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:39:22.160084 9293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39351
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35291
--tserver_master_addrs=127.0.98.62:42451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:39:22.161020 9293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:39:22.161942 9293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:39:22.171166 9299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:22.171198 9298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:22.171196 9301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:22.171749 9293 server_base.cc:1034] running on GCE node
I20260927 02:39:22.172205 9293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:39:22.172797 9293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:39:22.173965 9293 hybrid_clock.cc:648] HybridClock initialized: now 1790476762173946 us; error 29 us; skew 500 ppm
Sep 27 02:39:22 dist-test-slave-zv8z krb5kdc[8558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476762, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:39:22.176630 9293 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 02:39:22.177460 9293 webserver.cc:460] Webserver started at http://127.0.98.2:35291/ using document root <none> and password file <none>
I20260927 02:39:22.177954 9293 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:22.178017 9293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:22.181794 9293 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:39:22.183940 9308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:22.185155 9293 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:39:22.185271 9293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-1/wal
uuid: "68a6173a40704018b49cd71d19cc6d4a"
format_stamp: "Formatted at 2026-09-27 02:39:21 on dist-test-slave-zv8z"
server_key: "474a40247881f149f6c8a32bc4c73d3b"
server_key_iv: "1c6b7143607754fd7d55e1ed70aa983d"
server_key_version: "encryptionkey@0"
I20260927 02:39:22.185846 9293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:22.196429 9293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:39:22.199625 9293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:39:22.199802 9293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:39:22.200371 9293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:39:22.201180 9293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:39:22.201234 9293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:22.201277 9293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:39:22.201306 9293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:22.209818 9293 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39351
I20260927 02:39:22.209849 9421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39351 every 8 connection(s)
I20260927 02:39:22.210840 9293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:39:22.216035 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 9293
I20260927 02:39:22.216475 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33111
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42451
--webserver_port=37341
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 02:39:22 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:22 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 02:39:22.221745 9291 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:22.212620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60807 (local address 127.0.98.62:42451)
0927 02:39:22.212762 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:22.212766 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:22.213571 (+ 805us) server_negotiation.cc:407] Connection header received
0927 02:39:22.214918 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:22.214922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:22.214972 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:22.215050 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:22.216242 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:22.216834 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:22.217952 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:22.218169 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:22.218276 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:22.221620 (+ 3344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60807: BlockingRecv error: recv got EOF from 127.0.98.2:60807 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 02:39:22.222385 9424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:22.213003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.2:60807)
0927 02:39:22.213423 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:22.213466 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:22.214715 (+ 1249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:22.215238 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:22.215245 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:22.215597 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:22.216081 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:22.216097 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:22.216962 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:22.216965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:22.217838 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:22.217847 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:22.218326 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:22.218330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:22.218486 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:22.219051 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:22.219066 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:22.221481 (+ 2415us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":107,"threads_started":1}
W20260927 02:39:22.222815 9422 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42451 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:42451: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 02:39:22 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:22 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:39:22.227821 9424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:22.223259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.2:43691)
0927 02:39:22.223370 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:22.223385 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:22.223459 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:22.223679 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:22.223682 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:22.223817 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:22.223897 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:22.223902 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:22.224573 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:22.224576 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:22.225113 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:22.225120 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:22.225506 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:22.225510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:22.225590 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:22.226053 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:22.226063 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:22.227706 (+ 1643us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 02:39:22.228032 9291 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:22.223321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43691 (local address 127.0.98.62:42451)
0927 02:39:22.223422 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:22.223425 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:22.223438 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:39:22.223541 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:22.223544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:22.223588 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:22.223649 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:22.223995 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:22.224481 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:22.225212 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:22.225417 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:22.225499 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:22.227833 (+ 2334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43691: BlockingRecv error: recv got EOF from 127.0.98.2:43691 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 27 02:39:22 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:22 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:39:22.233266 9424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:22.228396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.2:44397)
0927 02:39:22.228538 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:22.228550 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:22.228622 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:22.228818 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:22.228821 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:22.228969 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:22.229048 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:22.229053 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:22.229692 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:22.229695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:22.230576 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:22.230585 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:22.230981 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:22.230985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:22.231061 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:22.231492 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:22.231507 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:22.233154 (+ 1647us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":84}
W20260927 02:39:22.233434 9422 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 02:39:22.233549 9291 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:22.228452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44397 (local address 127.0.98.62:42451)
0927 02:39:22.228543 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:22.228546 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:22.228587 (+ 41us) server_negotiation.cc:407] Connection header received
0927 02:39:22.228699 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:22.228701 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:22.228733 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:22.228791 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:22.229136 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:22.229596 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:22.230689 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:22.230894 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:22.230967 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:22.233274 (+ 2307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44397: BlockingRecv error: recv got EOF from 127.0.98.2:44397 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260927 02:39:22.324723 9425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:39:22.324987 9425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:39:22.325047 9425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:39:22.329030 9425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:39:22.329087 9425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:39:22.329166 9425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:39:22.334268 9425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33111
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=37341
--tserver_master_addrs=127.0.98.62:42451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:39:22.335237 9425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:39:22.336199 9425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:39:22.345417 9430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:22.345396 9431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:22.345450 9425 server_base.cc:1034] running on GCE node
W20260927 02:39:22.345398 9433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:22.346057 9425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:39:22.346791 9425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:39:22.347985 9425 hybrid_clock.cc:648] HybridClock initialized: now 1790476762347966 us; error 28 us; skew 500 ppm
Sep 27 02:39:22 dist-test-slave-zv8z krb5kdc[8558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476762, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:39:22.350533 9425 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 02:39:22.351331 9425 webserver.cc:460] Webserver started at http://127.0.98.3:37341/ using document root <none> and password file <none>
I20260927 02:39:22.351867 9425 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:22.351943 9425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:22.355747 9425 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:39:22.357695 9440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:22.358876 9425 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:39:22.358978 9425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-2/wal
uuid: "14870d4394664dfa951acda7dc1680d2"
format_stamp: "Formatted at 2026-09-27 02:39:21 on dist-test-slave-zv8z"
server_key: "8fe2be6b1ea8fc34923a9e991192c2cf"
server_key_iv: "02902c03e2ce08078800972e996fb3da"
server_key_version: "encryptionkey@0"
I20260927 02:39:22.359486 9425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:22.382001 9425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:39:22.385139 9425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:39:22.385352 9425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:39:22.385912 9425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:39:22.386778 9425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:39:22.386828 9425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:22.386879 9425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:39:22.386909 9425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:22.395581 9425 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33111
I20260927 02:39:22.395613 9553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33111 every 8 connection(s)
I20260927 02:39:22.396559 9425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:39:22.402383 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 9425
Sep 27 02:39:22 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:22 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 02:39:22.408325 9291 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:22.398402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60543 (local address 127.0.98.62:42451)
0927 02:39:22.398522 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:22.398527 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:22.399469 (+ 942us) server_negotiation.cc:407] Connection header received
0927 02:39:22.400800 (+ 1331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:22.400803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:22.400860 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:22.400941 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:22.402095 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:22.402854 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:22.404480 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:22.404746 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:22.404849 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:22.408205 (+ 3356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60543: BlockingRecv error: recv got EOF from 127.0.98.3:60543 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 02:39:22.409232 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:22.398835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.3:60543)
0927 02:39:22.399323 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:22.399362 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:22.400575 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:22.401137 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:22.401144 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:22.401484 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:22.401958 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:22.401971 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:22.402975 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:22.402978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:22.404351 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:22.404362 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:22.404894 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:22.404896 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:22.405040 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:22.405658 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:22.405675 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:22.408098 (+ 2423us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":160,"threads_started":1}
W20260927 02:39:22.409713 9554 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42451 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:42451: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260927 02:39:22.412055 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:22.403443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38036 (local address 127.0.98.62:42451)
0927 02:39:22.403683 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:22.403686 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:22.403698 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:39:22.403729 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:22.403731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:22.403765 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:22.403839 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:22.404662 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:22.405263 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:22.406170 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:22.406467 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:22.406558 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:22.407884 (+ 1326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:22.407891 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:22.407910 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:22.407942 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:22.409803 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:22.410236 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:22.410242 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:22.410270 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:22.410421 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:22.410800 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:22.410804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:22.410805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:22.411248 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:22.411445 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:22.411653 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 02:39:22.411889 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":93,"threads_started":1}
Sep 27 02:39:22 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:22 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:39:22.415160 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:22.410096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.3:36937)
0927 02:39:22.410287 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:22.410300 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:22.410369 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:22.410555 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:22.410557 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:22.410735 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:22.410831 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:22.410836 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:22.411754 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:22.411758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:22.412373 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:22.412382 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:22.412788 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:22.412792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:22.412883 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:22.413330 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:22.413340 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:22.414997 (+ 1657us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108}
W20260927 02:39:22.415198 9291 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:22.410227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36937 (local address 127.0.98.62:42451)
0927 02:39:22.410364 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:22.410368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:22.410380 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:39:22.410418 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:22.410420 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:22.410459 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:22.410527 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:22.410941 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:22.411648 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:22.412479 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:22.412692 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:22.412770 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:22.415113 (+ 2343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36937: BlockingRecv error: recv got EOF from 127.0.98.3:36937 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 27 02:39:22 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:22 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:39:22.419838 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:22.415626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.3:41555)
0927 02:39:22.415709 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:22.415720 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:22.415785 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:22.415995 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:22.415998 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:22.416169 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:22.416233 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:22.416237 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:22.416898 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:22.416901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:22.417415 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:22.417422 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:22.417811 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:22.417815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:22.417885 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:22.418306 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:22.418317 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:22.419756 (+ 1439us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":25}
W20260927 02:39:22.419952 9291 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:22.415702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41555 (local address 127.0.98.62:42451)
0927 02:39:22.415833 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:22.415837 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:22.415848 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:39:22.415877 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:22.415879 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:22.415913 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:22.415971 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:22.416329 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:22.416809 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:22.417518 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:22.417725 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:22.417803 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:22.419862 (+ 2059us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41555: BlockingRecv error: recv got EOF from 127.0.98.3:41555 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260927 02:39:22.420009 9554 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 02:39:23 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:23 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:39:23.068991 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:23.063794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.1:45873)
0927 02:39:23.064070 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:23.064084 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:23.064181 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:23.064402 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:23.064403 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:23.064608 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:23.064747 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:23.064754 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:23.065497 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:23.065499 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:23.066108 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:23.066117 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:23.066592 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:23.066594 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:23.066658 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:23.067102 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:23.067112 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:23.068838 (+ 1726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":78,"threads_started":1}
W20260927 02:39:23.069087 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:23.063890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45873 (local address 127.0.98.62:42451)
0927 02:39:23.064098 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:23.064101 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:23.064151 (+ 50us) server_negotiation.cc:407] Connection header received
0927 02:39:23.064270 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:23.064272 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:23.064314 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:23.064379 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:23.064825 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:23.065392 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:23.066279 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:23.066503 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:23.066582 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:23.068972 (+ 2390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45873: BlockingRecv error: recv got EOF from 127.0.98.1:45873 (error 108)
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":55,"threads_started":1}
Sep 27 02:39:23 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:23 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:39:23.239534 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:23.234070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.2:55151)
0927 02:39:23.234356 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:23.234370 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:23.234467 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:23.234726 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:23.234729 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:23.234909 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:23.235023 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:23.235029 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:23.235869 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:23.235872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:23.236491 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:23.236500 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:23.236961 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:23.236964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:23.237045 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:23.237559 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:23.237572 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:23.239406 (+ 1834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":78,"threads_started":1}
W20260927 02:39:23.239658 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:23.234163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55151 (local address 127.0.98.62:42451)
0927 02:39:23.234327 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:23.234331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:23.234433 (+ 102us) server_negotiation.cc:407] Connection header received
0927 02:39:23.234537 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:23.234539 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:23.234581 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:23.234652 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:23.235144 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:23.235731 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:23.236624 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:23.236846 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:23.236930 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:23.239542 (+ 2612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55151: BlockingRecv error: recv got EOF from 127.0.98.2:55151 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 27 02:39:23 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:23 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:39:23.425606 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:23.420554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.3:60329)
0927 02:39:23.420792 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:23.420806 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:23.420900 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:23.421125 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:23.421127 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:23.421333 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:23.421429 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:23.421435 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:23.422167 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:23.422169 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:23.422832 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:23.422842 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:23.423221 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:23.423223 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:23.423324 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:23.423792 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:23.423806 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:23.425487 (+ 1681us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":68,"threads_started":1}
W20260927 02:39:23.425729 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:23.420615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60329 (local address 127.0.98.62:42451)
0927 02:39:23.420739 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:23.420743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:23.420864 (+ 121us) server_negotiation.cc:407] Connection header received
0927 02:39:23.420968 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:23.420969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:23.421019 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:23.421086 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:23.421542 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:23.422065 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:23.422924 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:23.423132 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:23.423201 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:23.425614 (+ 2413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60329: BlockingRecv error: recv got EOF from 127.0.98.3:60329 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 02:39:24 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:24 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:39:24.075510 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:24.069941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.1:34079)
0927 02:39:24.070238 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:24.070277 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:24.070386 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:24.070657 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:24.070660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:24.070849 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:24.070954 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:24.070960 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:24.071786 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:24.071789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:24.072398 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:24.072407 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:24.072859 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:24.072862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:24.072934 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:24.073442 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:24.073454 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:24.075364 (+ 1910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":85,"threads_started":1}
W20260927 02:39:24.075615 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:24.069998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34079 (local address 127.0.98.62:42451)
0927 02:39:24.070285 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:24.070288 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:24.070344 (+ 56us) server_negotiation.cc:407] Connection header received
0927 02:39:24.070458 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:24.070460 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:24.070503 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:24.070574 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:24.071117 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:24.071657 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:24.072519 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:24.072740 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:24.072824 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:24.075500 (+ 2676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34079: BlockingRecv error: recv got EOF from 127.0.98.1:34079 (error 108)
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":113,"threads_started":1}
Sep 27 02:39:24 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:24 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:39:24.245510 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:24.240219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.2:48615)
0927 02:39:24.240536 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:24.240549 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:24.240642 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:24.240878 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:24.240880 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:24.241047 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:24.241140 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:24.241145 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:24.241847 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:24.241850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:24.242477 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:24.242487 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:24.242992 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:24.242995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:24.243056 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:24.243499 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:24.243510 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:24.245338 (+ 1828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":82,"threads_started":1}
W20260927 02:39:24.245575 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:24.240305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48615 (local address 127.0.98.62:42451)
0927 02:39:24.240425 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:24.240429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:24.240606 (+ 177us) server_negotiation.cc:407] Connection header received
0927 02:39:24.240718 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:24.240720 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:24.240774 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:24.240851 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:24.241233 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:24.241746 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:24.242617 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:24.242888 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:24.242970 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:24.245478 (+ 2508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48615: BlockingRecv error: recv got EOF from 127.0.98.2:48615 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 27 02:39:24 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:24 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:39:24.431316 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:24.426311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.3:41007)
0927 02:39:24.426557 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:24.426571 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:24.426683 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:24.426915 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:24.426918 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:24.427108 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:24.427199 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:24.427205 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:24.427908 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:24.427912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:24.428474 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:24.428482 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:24.428888 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:24.428891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:24.428967 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:24.429448 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:24.429462 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:24.431139 (+ 1677us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":78,"threads_started":1}
W20260927 02:39:24.431371 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:24.426429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41007 (local address 127.0.98.62:42451)
0927 02:39:24.426557 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:24.426562 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:24.426623 (+ 61us) server_negotiation.cc:407] Connection header received
0927 02:39:24.426766 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:24.426769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:24.426813 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:24.426899 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:24.427308 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:24.427811 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:24.428580 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:24.428794 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:24.428887 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:24.431274 (+ 2387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41007: BlockingRecv error: recv got EOF from 127.0.98.3:41007 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 27 02:39:25 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:25 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:39:25.085608 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:25.076352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.1:42693)
0927 02:39:25.076671 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:25.076686 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:25.076802 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:25.077083 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:25.077086 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:25.077290 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:25.077406 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:25.077415 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:25.078170 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:25.078173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:25.078779 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:25.078787 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:25.079177 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:25.079179 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:25.079234 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:25.083486 (+ 4252us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:25.083499 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:25.085455 (+ 1956us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":80,"threads_started":1}
W20260927 02:39:25.085728 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:25.076418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42693 (local address 127.0.98.62:42451)
0927 02:39:25.076680 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:25.076685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:25.076748 (+ 63us) server_negotiation.cc:407] Connection header received
0927 02:39:25.076899 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:25.076902 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:25.076955 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:25.077053 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:25.077519 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:25.078066 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:25.078870 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:25.079099 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:25.079170 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:25.085613 (+ 6443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42693: BlockingRecv error: recv got EOF from 127.0.98.1:42693 (error 108)
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":71,"threads_started":1}
Sep 27 02:39:25 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:25 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:39:25.252029 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:25.246403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.2:55595)
0927 02:39:25.246701 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:25.246715 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:25.246850 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:25.247102 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:25.247105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:25.247292 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:25.247463 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:25.247470 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:25.248226 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:25.248229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:25.248820 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:25.248829 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:25.249242 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:25.249245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:25.249322 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:25.249813 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:25.249825 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:25.251881 (+ 2056us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":101,"threads_started":1}
W20260927 02:39:25.252138 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:25.246465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55595 (local address 127.0.98.62:42451)
0927 02:39:25.246654 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:25.246658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:25.246786 (+ 128us) server_negotiation.cc:407] Connection header received
0927 02:39:25.246951 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:25.246954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:25.247005 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:25.247073 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:25.247575 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:25.248122 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:25.248927 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:25.249145 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:25.249231 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:25.252034 (+ 2803us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55595: BlockingRecv error: recv got EOF from 127.0.98.2:55595 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 27 02:39:25 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:25 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:39:25.437332 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:25.432130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.3:60015)
0927 02:39:25.432385 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:25.432399 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:25.432509 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:25.432751 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:25.432754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:25.432955 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:25.433048 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:25.433054 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:25.433797 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:25.433800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:25.434413 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:25.434421 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:25.434833 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:25.434836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:25.434912 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:25.435361 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:25.435372 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:25.437159 (+ 1787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":78,"threads_started":1}
W20260927 02:39:25.437400 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:25.432241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60015 (local address 127.0.98.62:42451)
0927 02:39:25.432372 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:25.432377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:25.432464 (+ 87us) server_negotiation.cc:407] Connection header received
0927 02:39:25.432605 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:25.432608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:25.432656 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:25.432721 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:25.433158 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:25.433689 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:25.434516 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:25.434739 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:25.434823 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:25.437304 (+ 2481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60015: BlockingRecv error: recv got EOF from 127.0.98.3:60015 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 27 02:39:26 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:26 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:39:26.091833 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:26.086453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.1:55057)
0927 02:39:26.086737 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:26.086752 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:26.086857 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:26.087099 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:26.087101 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:26.087271 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:26.087362 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:26.087368 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:26.088105 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:26.088108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:26.088793 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:26.088804 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:26.089345 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:26.089349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:26.089436 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:26.089956 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:26.089967 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:26.091721 (+ 1754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":91,"threads_started":1}
W20260927 02:39:26.091989 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:26.086526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55057 (local address 127.0.98.62:42451)
0927 02:39:26.086762 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:26.086765 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:26.086843 (+ 78us) server_negotiation.cc:407] Connection header received
0927 02:39:26.086931 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:26.086934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:26.086998 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:26.087075 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:26.087455 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:26.088004 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:26.088912 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:26.089253 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:26.089334 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:26.091857 (+ 2523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55057: BlockingRecv error: recv got EOF from 127.0.98.1:55057 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":67,"threads_started":1}
Sep 27 02:39:26 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:26 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:39:26.259289 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:26.252868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.2:57907)
0927 02:39:26.253181 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:26.253194 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:26.253295 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:26.253569 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:26.253571 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:26.253757 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:26.253867 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:26.253873 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:26.254952 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:26.254955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:26.255676 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:26.255687 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:26.256294 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:26.256297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:26.256371 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:26.256993 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:26.257005 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:26.259113 (+ 2108us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":80,"threads_started":1}
W20260927 02:39:26.259347 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:26.252978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57907 (local address 127.0.98.62:42451)
0927 02:39:26.253134 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:26.253139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:26.253262 (+ 123us) server_negotiation.cc:407] Connection header received
0927 02:39:26.253375 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:26.253379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:26.253434 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:26.253529 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:26.253972 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:26.254845 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:26.255786 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:26.256152 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:26.256258 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:26.259253 (+ 2995us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57907: BlockingRecv error: recv got EOF from 127.0.98.2:57907 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 27 02:39:26 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:26 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:39:26.443492 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:26.438120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.3:58589)
0927 02:39:26.438408 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:26.438422 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:26.438533 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:26.438803 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:26.438807 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:26.439059 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:26.439177 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:26.439183 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:26.440032 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:26.440035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:26.440609 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:26.440619 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:26.441048 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:26.441051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:26.441144 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:26.441670 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:26.441684 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:26.443373 (+ 1689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":80,"threads_started":1}
W20260927 02:39:26.443611 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:26.438223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58589 (local address 127.0.98.62:42451)
0927 02:39:26.438378 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:26.438404 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:26.438490 (+ 86us) server_negotiation.cc:407] Connection header received
0927 02:39:26.438624 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:26.438627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:26.438674 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:26.438752 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:26.439279 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:26.439916 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:26.440718 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:26.440928 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:26.441003 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:26.443498 (+ 2495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58589: BlockingRecv error: recv got EOF from 127.0.98.3:58589 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 02:39:27 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:27 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:39:27.098129 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:27.092625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.1:42897)
0927 02:39:27.092900 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:27.092917 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:27.093034 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:27.093304 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:27.093307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:27.093524 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:27.093662 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:27.093669 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:27.094490 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:27.094493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:27.095127 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:27.095136 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:27.095555 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:27.095558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:27.095620 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:27.096144 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:27.096155 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:27.097975 (+ 1820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":89,"threads_started":1}
W20260927 02:39:27.098280 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:27.092697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42897 (local address 127.0.98.62:42451)
0927 02:39:27.092927 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:27.092932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:27.092990 (+ 58us) server_negotiation.cc:407] Connection header received
0927 02:39:27.093124 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:27.093127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:27.093182 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:27.093286 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:27.093785 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:27.094363 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:27.095241 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:27.095469 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:27.095546 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:27.098132 (+ 2586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42897: BlockingRecv error: recv got EOF from 127.0.98.1:42897 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":70,"threads_started":1}
Sep 27 02:39:27 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:27 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:39:27.265919 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:27.260196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.2:37065)
0927 02:39:27.260482 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:27.260495 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:27.260583 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:27.260837 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:27.260839 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:27.261028 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:27.261121 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:27.261127 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:27.261949 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:27.261952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:27.262636 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:27.262646 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:27.263143 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:27.263147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:27.263217 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:27.263774 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:27.263788 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:27.265750 (+ 1962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":79,"threads_started":1}
W20260927 02:39:27.266160 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:27.260404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37065 (local address 127.0.98.62:42451)
0927 02:39:27.260536 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:27.260540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:27.260553 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:39:27.260677 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:27.260679 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:27.260724 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:27.260792 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:27.261251 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:27.261828 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:27.262788 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:27.263013 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:27.263109 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:27.265943 (+ 2834us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37065: BlockingRecv error: recv got EOF from 127.0.98.2:37065 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 02:39:27 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:27 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:39:27.449374 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:27.444196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.3:56317)
0927 02:39:27.444444 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:27.444457 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:27.444549 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:27.444768 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:27.444770 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:27.444939 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:27.445031 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:27.445035 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:27.445782 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:27.445784 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:27.446394 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:27.446404 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:27.446940 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:27.446943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:27.447009 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:27.447479 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:27.447489 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:27.449245 (+ 1756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":79,"threads_started":1}
W20260927 02:39:27.449501 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:27.444295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56317 (local address 127.0.98.62:42451)
0927 02:39:27.444409 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:27.444413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:27.444572 (+ 159us) server_negotiation.cc:407] Connection header received
0927 02:39:27.444615 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:27.444617 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:27.444664 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:27.444737 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:27.445160 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:27.445683 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:27.446519 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:27.446805 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:27.446892 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:27.449386 (+ 2494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56317: BlockingRecv error: recv got EOF from 127.0.98.3:56317 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 27 02:39:28 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:28 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:39:28.104029 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:28.098905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.1:34981)
0927 02:39:28.099173 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:28.099186 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:28.099285 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:28.099505 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:28.099507 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:28.099674 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:28.099766 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:28.099771 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:28.100527 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:28.100529 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:28.101099 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:28.101106 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:28.101506 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:28.101509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:28.101564 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:28.102110 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:28.102121 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:28.103873 (+ 1752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":93,"threads_started":1}
W20260927 02:39:28.104126 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:28.098935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34981 (local address 127.0.98.62:42451)
0927 02:39:28.099192 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:28.099196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:28.099242 (+ 46us) server_negotiation.cc:407] Connection header received
0927 02:39:28.099362 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:28.099364 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:28.099406 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:28.099480 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:28.099889 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:28.100440 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:28.101191 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:28.101424 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:28.101499 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:28.104010 (+ 2511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34981: BlockingRecv error: recv got EOF from 127.0.98.1:34981 (error 108)
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":75,"threads_started":1}
Sep 27 02:39:28 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:28 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:39:28.271752 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:28.266711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.2:37601)
0927 02:39:28.266965 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:28.266980 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:28.267094 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:28.267367 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:28.267370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:28.267610 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:28.267725 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:28.267732 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:28.268543 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:28.268546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:28.269103 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:28.269111 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:28.269494 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:28.269497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:28.269555 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:28.269989 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:28.270000 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:28.271640 (+ 1640us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":79,"threads_started":1}
W20260927 02:39:28.271924 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:28.266781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37601 (local address 127.0.98.62:42451)
0927 02:39:28.266927 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:28.266964 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:28.267049 (+ 85us) server_negotiation.cc:407] Connection header received
0927 02:39:28.267195 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:28.267198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:28.267254 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:28.267329 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:28.267840 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:28.268440 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:28.269195 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:28.269404 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:28.269475 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:28.271808 (+ 2333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37601: BlockingRecv error: recv got EOF from 127.0.98.2:37601 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 27 02:39:28 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:28 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:39:28.455219 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:28.450112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.3:54007)
0927 02:39:28.450378 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:28.450390 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:28.450487 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:28.450709 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:28.450711 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:28.450876 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:28.450978 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:28.450984 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:28.451658 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:28.451660 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:28.452258 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:28.452265 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:28.452643 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:28.452646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:28.452709 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:28.453209 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:28.453221 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:28.455074 (+ 1853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":72,"threads_started":1}
W20260927 02:39:28.455307 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:28.450198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54007 (local address 127.0.98.62:42451)
0927 02:39:28.450355 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:28.450360 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:28.450468 (+ 108us) server_negotiation.cc:407] Connection header received
0927 02:39:28.450562 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:28.450564 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:28.450609 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:28.450676 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:28.451065 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:28.451579 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:28.452342 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:28.452552 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:28.452620 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:28.455207 (+ 2587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54007: BlockingRecv error: recv got EOF from 127.0.98.3:54007 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 02:39:29 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:29 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:39:29.115303 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:29.104744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.1:58777)
0927 02:39:29.105118 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:29.105134 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:29.105238 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:29.105509 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:29.105512 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:29.105678 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:29.105773 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:29.105778 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:29.106491 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:29.106493 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:29.107069 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:29.107077 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:29.107473 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:29.107476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:29.107535 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:29.113264 (+ 5729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:29.113278 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:29.115154 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":77,"threads_started":1}
W20260927 02:39:29.115401 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:29.104811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58777 (local address 127.0.98.62:42451)
0927 02:39:29.105144 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:29.105148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:29.105183 (+ 35us) server_negotiation.cc:407] Connection header received
0927 02:39:29.105343 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:29.105345 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:29.105393 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:29.105475 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:29.105858 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:29.106412 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:29.107163 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:29.107392 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:29.107466 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:29.115287 (+ 7821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58777: BlockingRecv error: recv got EOF from 127.0.98.1:58777 (error 108)
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":75,"threads_started":1}
Sep 27 02:39:29 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:29 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:39:29.278053 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:29.272508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.2:60761)
0927 02:39:29.272796 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:29.272809 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:29.272955 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:29.273242 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:29.273244 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:29.273422 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:29.273519 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:29.273524 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:29.274277 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:29.274281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:29.274854 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:29.274863 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:29.275352 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:29.275355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:29.275424 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:29.275921 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:29.275933 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:29.277897 (+ 1964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":86,"threads_started":1}
W20260927 02:39:29.278131 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:29.272583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60761 (local address 127.0.98.62:42451)
0927 02:39:29.272724 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:29.272728 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:29.272867 (+ 139us) server_negotiation.cc:407] Connection header received
0927 02:39:29.273059 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:29.273061 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:29.273106 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:29.273176 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:29.273614 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:29.274146 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:29.275001 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:29.275228 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:29.275315 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:29.278028 (+ 2713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60761: BlockingRecv error: recv got EOF from 127.0.98.2:60761 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 02:39:29 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:29 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:39:29.461249 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:29.455991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.3:51099)
0927 02:39:29.456355 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:29.456368 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:29.456476 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:29.456725 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:29.456728 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:29.456911 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:29.457008 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:29.457014 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:29.457738 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:29.457741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:29.458395 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:29.458404 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:29.458850 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:29.458854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:29.458933 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:29.459430 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:29.459441 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:29.461092 (+ 1651us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":104,"threads_started":1}
W20260927 02:39:29.461334 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:29.456118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51099 (local address 127.0.98.62:42451)
0927 02:39:29.456273 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:29.456277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:29.456441 (+ 164us) server_negotiation.cc:407] Connection header received
0927 02:39:29.456568 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:29.456572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:29.456619 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:29.456685 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:29.457120 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:29.457639 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:29.458526 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:29.458749 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:29.458836 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:29.461233 (+ 2397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51099: BlockingRecv error: recv got EOF from 127.0.98.3:51099 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 27 02:39:30 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:30 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:39:30.124864 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:30.116057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.1:35167)
0927 02:39:30.116371 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:30.116386 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:30.116475 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:30.116681 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:30.116683 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:30.116849 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:30.116944 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:30.116949 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:30.117826 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:30.117829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:30.118441 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:30.118452 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:30.118862 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:30.118865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:30.118923 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:30.122857 (+ 3934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:30.122870 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:30.124744 (+ 1874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":79,"threads_started":1}
W20260927 02:39:30.125134 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:30.116208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35167 (local address 127.0.98.62:42451)
0927 02:39:30.116419 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:30.116423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:30.116435 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:39:30.116541 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:30.116543 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:30.116582 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:30.116653 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:30.117088 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:30.117720 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:30.118546 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:30.118775 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:30.118855 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:30.124916 (+ 6061us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35167: BlockingRecv error: recv got EOF from 127.0.98.1:35167 (error 108)
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":72,"threads_started":1}
Sep 27 02:39:30 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:30 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:39:30.284142 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:30.278773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.2:49361)
0927 02:39:30.279038 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:30.279051 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:30.279158 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:30.279414 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:30.279416 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:30.279584 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:30.279674 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:30.279679 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:30.280659 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:30.280663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:30.281238 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:30.281247 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:30.281729 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:30.281732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:30.281794 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:30.282301 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:30.282312 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:30.284024 (+ 1712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":79,"threads_started":1}
W20260927 02:39:30.284284 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:30.278879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49361 (local address 127.0.98.62:42451)
0927 02:39:30.279195 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:30.279199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:30.279231 (+ 32us) server_negotiation.cc:407] Connection header received
0927 02:39:30.279269 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:30.279272 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:30.279315 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:30.279373 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:30.279827 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:30.280552 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:30.281377 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:30.281612 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:30.281698 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:30.284164 (+ 2466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49361: BlockingRecv error: recv got EOF from 127.0.98.2:49361 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 02:39:30 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:30 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:39:30.467769 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:30.462155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.3:44885)
0927 02:39:30.462426 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:30.462439 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:30.462526 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:30.462792 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:30.462794 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:30.462961 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:30.463050 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:30.463056 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:30.463916 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:30.463920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:30.464692 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:30.464704 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:30.465327 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:30.465331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:30.465413 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:30.465845 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:30.465856 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:30.467653 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":111,"threads_started":1}
W20260927 02:39:30.467980 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:30.462347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44885 (local address 127.0.98.62:42451)
0927 02:39:30.462462 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:30.462466 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:30.462483 (+ 17us) server_negotiation.cc:407] Connection header received
0927 02:39:30.462613 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:30.462615 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:30.462662 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:30.462738 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:30.463186 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:30.463778 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:30.464820 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:30.465203 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:30.465305 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:30.467797 (+ 2492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44885: BlockingRecv error: recv got EOF from 127.0.98.3:44885 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 27 02:39:31 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:31 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:39:31.131198 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:31.125661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.1:35515)
0927 02:39:31.125937 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:31.125950 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:31.126041 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:31.126362 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:31.126364 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:31.126529 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:31.126628 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:31.126633 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:31.127358 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:31.127360 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:31.127906 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:31.127913 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:31.128411 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:31.128413 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:31.128465 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:31.128956 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:31.128969 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:31.131076 (+ 2107us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":77,"threads_started":1}
W20260927 02:39:31.131304 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:31.125757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35515 (local address 127.0.98.62:42451)
0927 02:39:31.125995 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:31.125999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:31.126012 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:39:31.126121 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:31.126123 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:31.126164 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:31.126314 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:31.126720 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:31.127272 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:31.128023 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:31.128321 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:31.128410 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:31.131188 (+ 2778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35515: BlockingRecv error: recv got EOF from 127.0.98.1:35515 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":76,"threads_started":1}
Sep 27 02:39:31 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:31 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:39:31.290162 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:31.284845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.2:32845)
0927 02:39:31.285080 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:31.285092 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:31.285184 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:31.285429 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:31.285431 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:31.285591 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:31.285686 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:31.285691 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:31.286491 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:31.286495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:31.287055 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:31.287064 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:31.287506 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:31.287509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:31.287574 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:31.288081 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:31.288093 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:31.290042 (+ 1949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":71,"threads_started":1}
W20260927 02:39:31.290360 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:31.284898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32845 (local address 127.0.98.62:42451)
0927 02:39:31.285050 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:31.285054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:31.285138 (+ 84us) server_negotiation.cc:407] Connection header received
0927 02:39:31.285245 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:31.285247 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:31.285288 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:31.285401 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:31.285771 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:31.286320 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:31.287162 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:31.287404 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:31.287485 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:31.290175 (+ 2690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:32845: BlockingRecv error: recv got EOF from 127.0.98.2:32845 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 02:39:31 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:31 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:39:31.473798 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:31.468429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.3:40347)
0927 02:39:31.468670 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:31.468684 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:31.468791 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:31.469033 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:31.469036 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:31.469225 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:31.469315 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:31.469321 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:31.470075 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:31.470079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:31.470749 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:31.470758 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:31.471196 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:31.471199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:31.471284 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:31.471835 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:31.471847 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:31.473673 (+ 1826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":71,"threads_started":1}
W20260927 02:39:31.473927 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:31.468483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40347 (local address 127.0.98.62:42451)
0927 02:39:31.468604 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:31.468608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:31.468752 (+ 144us) server_negotiation.cc:407] Connection header received
0927 02:39:31.468883 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:31.468886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:31.468935 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:31.468996 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:31.469432 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:31.469972 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:31.470871 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:31.471085 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:31.471168 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:31.473802 (+ 2634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40347: BlockingRecv error: recv got EOF from 127.0.98.3:40347 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 27 02:39:32 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:32 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:39:32.137097 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:32.131952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.1:57919)
0927 02:39:32.132206 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:32.132219 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:32.132313 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:32.132537 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:32.132539 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:32.132713 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:32.132835 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:32.132841 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:32.133563 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:32.133566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:32.134143 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:32.134151 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:32.134616 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:32.134619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:32.134680 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:32.135152 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:32.135162 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:32.136949 (+ 1787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":70,"threads_started":1}
W20260927 02:39:32.137188 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:32.132027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57919 (local address 127.0.98.62:42451)
0927 02:39:32.132237 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:32.132240 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:32.132267 (+ 27us) server_negotiation.cc:407] Connection header received
0927 02:39:32.132381 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:32.132383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:32.132422 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:32.132501 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:32.132932 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:32.133474 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:32.134243 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:32.134516 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:32.134593 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:32.137072 (+ 2479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57919: BlockingRecv error: recv got EOF from 127.0.98.1:57919 (error 108)
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":68,"threads_started":1}
Sep 27 02:39:32 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:32 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:39:32.296111 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:32.290903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.2:56115)
0927 02:39:32.291186 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:32.291200 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:32.291293 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:32.291525 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:32.291527 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:32.291698 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:32.291801 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:32.291806 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:32.292526 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:32.292529 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:32.293181 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:32.293192 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:32.293600 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:32.293603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:32.293659 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:32.294126 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:32.294138 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:32.295999 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":89,"threads_started":1}
W20260927 02:39:32.296236 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:32.290969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56115 (local address 127.0.98.62:42451)
0927 02:39:32.291107 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:32.291112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:32.291263 (+ 151us) server_negotiation.cc:407] Connection header received
0927 02:39:32.291364 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:32.291366 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:32.291410 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:32.291475 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:32.291896 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:32.292422 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:32.293293 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:32.293515 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:32.293591 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:32.296121 (+ 2530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56115: BlockingRecv error: recv got EOF from 127.0.98.2:56115 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 02:39:32 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:32 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:39:32.479573 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:32.474576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.3:45495)
0927 02:39:32.474817 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:32.474833 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:32.474928 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:32.475156 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:32.475159 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:32.475340 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:32.475427 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:32.475433 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:32.476160 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:32.476163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:32.476727 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:32.476735 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:32.477162 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:32.477166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:32.477260 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:32.477701 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:32.477712 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:32.479458 (+ 1746us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":69,"threads_started":1}
W20260927 02:39:32.479714 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:32.474723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45495 (local address 127.0.98.62:42451)
0927 02:39:32.474856 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:32.474860 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:32.474877 (+ 17us) server_negotiation.cc:407] Connection header received
0927 02:39:32.475011 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:32.475014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:32.475063 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:32.475125 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:32.475535 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:32.476060 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:32.476834 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:32.477064 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:32.477148 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:32.479596 (+ 2448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45495: BlockingRecv error: recv got EOF from 127.0.98.3:45495 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 27 02:39:33 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:33 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:39:33.143347 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:33.138015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.1:56085)
0927 02:39:33.138404 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:33.138417 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:33.138521 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:33.138790 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:33.138794 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:33.138983 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:33.139080 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:33.139086 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:33.139833 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:33.139836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:33.140393 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:33.140401 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:33.140885 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:33.140888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:33.140963 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:33.141512 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:33.141526 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:33.143233 (+ 1707us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":73,"threads_started":1}
W20260927 02:39:33.143476 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:33.138143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56085 (local address 127.0.98.62:42451)
0927 02:39:33.138385 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:33.138390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:33.138488 (+ 98us) server_negotiation.cc:407] Connection header received
0927 02:39:33.138604 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:33.138608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:33.138663 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:33.138751 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:33.139171 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:33.139731 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:33.140531 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:33.140750 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:33.140826 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:33.143358 (+ 2532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56085: BlockingRecv error: recv got EOF from 127.0.98.1:56085 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":71,"threads_started":1}
Sep 27 02:39:33 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:33 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:39:33.302194 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:33.296881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.2:37883)
0927 02:39:33.297136 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:33.297154 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:33.297256 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:33.297488 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:33.297490 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:33.297690 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:33.297783 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:33.297789 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:33.298649 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:33.298652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:33.299223 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:33.299231 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:33.299693 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:33.299697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:33.299784 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:33.300300 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:33.300311 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:33.302079 (+ 1768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":72,"threads_started":1}
W20260927 02:39:33.302387 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:33.296946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37883 (local address 127.0.98.62:42451)
0927 02:39:33.297095 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:33.297099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:33.297221 (+ 122us) server_negotiation.cc:407] Connection header received
0927 02:39:33.297331 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:33.297334 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:33.297380 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:33.297457 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:33.297935 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:33.298520 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:33.299328 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:33.299557 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:33.299636 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:33.302270 (+ 2634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37883: BlockingRecv error: recv got EOF from 127.0.98.2:37883 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 27 02:39:33 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:33 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:39:33.486016 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:33.480312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.3:41359)
0927 02:39:33.480671 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:33.480688 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:33.480809 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:33.481082 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:33.481084 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:33.481253 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:33.481348 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:33.481354 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:33.482099 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:33.482103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:33.482774 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:33.482784 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:33.483287 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:33.483291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:33.483374 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:33.484009 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:33.484025 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:33.485900 (+ 1875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":85,"threads_started":1}
W20260927 02:39:33.486191 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:33.480409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41359 (local address 127.0.98.62:42451)
0927 02:39:33.480557 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:33.480562 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:33.480776 (+ 214us) server_negotiation.cc:407] Connection header received
0927 02:39:33.480907 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:33.480910 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:33.480963 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:33.481040 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:33.481437 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:33.481980 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:33.482881 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:33.483163 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:33.483255 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:33.486043 (+ 2788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41359: BlockingRecv error: recv got EOF from 127.0.98.3:41359 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 02:39:34 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:34 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:39:34.149559 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:34.144041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.1:50297)
0927 02:39:34.144331 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:34.144348 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:34.144457 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:34.144735 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:34.144738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:34.144961 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:34.145083 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:34.145091 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:34.145948 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:34.145950 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:34.146585 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:34.146595 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:34.147041 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:34.147045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:34.147125 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:34.147654 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:34.147666 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:34.149443 (+ 1777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":91,"threads_started":1}
W20260927 02:39:34.149674 9604 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:34.144176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50297 (local address 127.0.98.62:42451)
0927 02:39:34.144441 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:34.144445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:34.144458 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:39:34.144551 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:34.144554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:34.144606 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:34.144695 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:34.145221 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:34.145827 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:34.146699 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:34.146918 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:34.146995 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:34.149567 (+ 2572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50297: BlockingRecv error: recv got EOF from 127.0.98.1:50297 (error 108)
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":87,"threads_started":1}
Sep 27 02:39:34 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:34 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:39:34.308055 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:34.302966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.2:58945)
0927 02:39:34.303262 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:34.303276 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:34.303390 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:34.303631 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:34.303634 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:34.303842 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:34.303936 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:34.303942 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:34.304663 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:34.304667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:34.305254 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:34.305263 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:34.305681 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:34.305684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:34.305761 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:34.306310 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:34.306322 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:34.307943 (+ 1621us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":83,"threads_started":1}
W20260927 02:39:34.308173 9604 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:34.303084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58945 (local address 127.0.98.62:42451)
0927 02:39:34.303225 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:34.303230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:34.303349 (+ 119us) server_negotiation.cc:407] Connection header received
0927 02:39:34.303480 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:34.303483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:34.303527 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:34.303592 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:34.304049 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:34.304561 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:34.305366 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:34.305582 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:34.305665 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:34.308071 (+ 2406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58945: BlockingRecv error: recv got EOF from 127.0.98.2:58945 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 02:39:34 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:34 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:39:34.492272 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:34.486747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.3:35107)
0927 02:39:34.487043 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:34.487059 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:34.487176 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:34.487427 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:34.487430 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:34.487624 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:34.487766 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:34.487772 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:34.488512 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:34.488517 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:34.489077 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:34.489086 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:34.489539 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:34.489543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:34.489621 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:34.490086 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:34.490097 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:34.492126 (+ 2029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":85,"threads_started":1}
W20260927 02:39:34.492383 9604 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:34.486845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35107 (local address 127.0.98.62:42451)
0927 02:39:34.487021 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:34.487026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:34.487133 (+ 107us) server_negotiation.cc:407] Connection header received
0927 02:39:34.487275 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:34.487278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:34.487326 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:34.487389 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:34.487885 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:34.488410 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:34.489189 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:34.489436 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:34.489523 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:34.492284 (+ 2761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35107: BlockingRecv error: recv got EOF from 127.0.98.3:35107 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 27 02:39:35 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:35 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:39:35.160255 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:35.150341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.1:52531)
0927 02:39:35.150622 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:35.150635 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:35.150730 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:35.151016 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:35.151018 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:35.151187 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:35.151307 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:35.151313 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:35.152096 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:35.152099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:35.152666 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:35.152674 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:35.153111 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:35.153114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:35.153180 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:35.158195 (+ 5015us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:35.158210 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:35.160073 (+ 1863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":82,"threads_started":1}
W20260927 02:39:35.160354 9608 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:35.150423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52531 (local address 127.0.98.62:42451)
0927 02:39:35.150716 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:35.150720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:35.150735 (+ 15us) server_negotiation.cc:407] Connection header received
0927 02:39:35.150815 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:35.150817 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:35.150861 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:35.150950 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:35.151437 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:35.151986 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:35.152780 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:35.153000 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:35.153078 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:35.160232 (+ 7154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52531: BlockingRecv error: recv got EOF from 127.0.98.1:52531 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":72,"threads_started":1}
Sep 27 02:39:35 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:35 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:39:35.314198 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:35.308769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.2:59123)
0927 02:39:35.309100 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:35.309115 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:35.309232 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:35.309495 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:35.309498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:35.309690 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:35.309782 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:35.309788 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:35.310642 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:35.310645 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:35.311204 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:35.311212 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:35.311634 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:35.311638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:35.311717 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:35.312223 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:35.312242 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:35.314050 (+ 1808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":139,"threads_started":1}
W20260927 02:39:35.314327 9608 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:35.308940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59123 (local address 127.0.98.62:42451)
0927 02:39:35.309102 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:35.309108 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:35.309191 (+ 83us) server_negotiation.cc:407] Connection header received
0927 02:39:35.309336 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:35.309339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:35.309389 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:35.309452 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:35.309925 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:35.310540 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:35.311308 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:35.311535 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:35.311618 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:35.314192 (+ 2574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59123: BlockingRecv error: recv got EOF from 127.0.98.2:59123 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 27 02:39:35 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:35 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:39:35.498451 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:35.493173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.3:58007)
0927 02:39:35.493429 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:35.493442 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:35.493551 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:35.493842 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:35.493845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:35.494055 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:35.494147 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:35.494153 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:35.494912 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:35.494916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:35.495492 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:35.495500 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:35.495963 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:35.495967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:35.496045 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:35.496527 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:35.496538 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:35.498293 (+ 1755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":93,"threads_started":1}
W20260927 02:39:35.498581 9608 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:35.493224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58007 (local address 127.0.98.62:42451)
0927 02:39:35.493378 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:35.493383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:35.493506 (+ 123us) server_negotiation.cc:407] Connection header received
0927 02:39:35.493635 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:35.493638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:35.493685 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:35.493806 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:35.494280 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:35.494810 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:35.495599 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:35.495859 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:35.495947 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:35.498462 (+ 2515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58007: BlockingRecv error: recv got EOF from 127.0.98.3:58007 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 02:39:36 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:36 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:39:36.166165 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:36.160997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.1:34267)
0927 02:39:36.161239 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:36.161251 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:36.161336 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:36.161576 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:36.161578 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:36.161740 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:36.161835 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:36.161840 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:36.162642 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:36.162644 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:36.163214 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:36.163221 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:36.163640 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:36.163644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:36.163722 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:36.164154 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:36.164165 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:36.166005 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":73,"threads_started":1}
W20260927 02:39:36.166244 9612 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:36.161068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34267 (local address 127.0.98.62:42451)
0927 02:39:36.161323 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:36.161326 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:36.161338 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:39:36.161401 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:36.161403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:36.161461 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:36.161533 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:36.161928 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:36.162513 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:36.163303 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:36.163540 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:36.163615 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:36.166129 (+ 2514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34267: BlockingRecv error: recv got EOF from 127.0.98.1:34267 (error 108)
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":62,"threads_started":1}
Sep 27 02:39:36 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:36 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:39:36.320573 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:36.315128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.2:46435)
0927 02:39:36.315407 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:36.315421 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:36.315564 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:36.315826 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:36.315829 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:36.316056 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:36.316172 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:36.316179 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:36.317023 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:36.317027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:36.317640 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:36.317648 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:36.318094 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:36.318098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:36.318172 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:36.318672 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:36.318683 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:36.320402 (+ 1719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":87,"threads_started":1}
W20260927 02:39:36.320655 9612 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:36.315262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46435 (local address 127.0.98.62:42451)
0927 02:39:36.315402 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:36.315406 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:36.315518 (+ 112us) server_negotiation.cc:407] Connection header received
0927 02:39:36.315655 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:36.315658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:36.315724 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:36.315788 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:36.316283 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:36.316920 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:36.317744 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:36.317995 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:36.318081 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:36.320548 (+ 2467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46435: BlockingRecv error: recv got EOF from 127.0.98.2:46435 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 02:39:36 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:36 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:39:36.504302 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:36.499247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.3:51207)
0927 02:39:36.499501 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:36.499522 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:36.499619 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:36.499836 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:36.499838 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:36.500005 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:36.500102 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:36.500108 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:36.500779 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:36.500781 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:36.501325 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:36.501332 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:36.501789 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:36.501792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:36.501861 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:36.502425 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:36.502437 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:36.504182 (+ 1745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":84,"threads_started":1}
W20260927 02:39:36.504415 9612 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:36.499323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51207 (local address 127.0.98.62:42451)
0927 02:39:36.499433 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:36.499437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:36.499582 (+ 145us) server_negotiation.cc:407] Connection header received
0927 02:39:36.499689 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:36.499692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:36.499737 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:36.499807 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:36.500189 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:36.500698 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:36.501421 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:36.501649 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:36.501764 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:36.504314 (+ 2550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51207: BlockingRecv error: recv got EOF from 127.0.98.3:51207 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 27 02:39:37 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:37 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:39:37.172228 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:37.167060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.1:33053)
0927 02:39:37.167303 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:37.167316 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:37.167396 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:37.167623 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:37.167625 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:37.167783 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:37.167872 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:37.167877 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:37.168620 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:37.168622 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:37.169186 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:37.169193 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:37.169597 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:37.169600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:37.169663 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:37.170121 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:37.170133 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:37.172108 (+ 1975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":68,"threads_started":1}
W20260927 02:39:37.172343 9616 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:37.167147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33053 (local address 127.0.98.62:42451)
0927 02:39:37.167352 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:37.167355 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:37.167368 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:39:37.167468 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:37.167470 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:37.167517 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:37.167618 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:37.167964 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:37.168514 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:37.169274 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:37.169505 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:37.169578 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:37.172221 (+ 2643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33053: BlockingRecv error: recv got EOF from 127.0.98.1:33053 (error 108)
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":63,"threads_started":1}
Sep 27 02:39:37 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:39:37 dist-test-slave-zv8z krb5kdc[8558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:39:37.327478 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:37.321550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.2:52651)
0927 02:39:37.321838 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:37.321850 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:37.321970 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:37.322240 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:37.322243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:37.322483 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:37.322593 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:37.322599 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:37.323436 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:37.323439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:37.324051 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:37.324059 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:37.324630 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:37.324634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:37.324698 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:37.325234 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:37.325246 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:37.327296 (+ 2050us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42451: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":103,"threads_started":1}
W20260927 02:39:37.327559 9616 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:37.321650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52651 (local address 127.0.98.62:42451)
0927 02:39:37.321775 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:37.321780 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:37.321939 (+ 159us) server_negotiation.cc:407] Connection header received
0927 02:39:37.322064 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:37.322067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:37.322120 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:37.322202 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:37.322714 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:37.323333 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:37.324179 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:37.324509 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:37.324617 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:37.327428 (+ 2811us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52651: BlockingRecv error: recv got EOF from 127.0.98.2:52651 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 02:39:37.413442 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 9141
I20260927 02:39:37.418143 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 9293
I20260927 02:39:37.422436 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 9425
I20260927 02:39:37.426743 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 9071
2026-09-27T02:39:37Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17808 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T02:39:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:39:37Z Disabled control of system clock
I20260927 02:39:37.453986 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36037
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36037
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 02:39:37.560106 9621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:39:37.560377 9621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:39:37.560410 9621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:39:37.564525 9621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:39:37.564589 9621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:39:37.564603 9621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:39:37.564620 9621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:39:37.564633 9621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:39:37.569717 9621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36037
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:39:37.570801 9621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:39:37.572146 9621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:39:37.580724 9627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:37.580829 9621 server_base.cc:1034] running on GCE node
W20260927 02:39:37.580865 9629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:37.581017 9626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:37.581563 9621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:39:37.582662 9621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:39:37.583812 9621 hybrid_clock.cc:648] HybridClock initialized: now 1790476777583792 us; error 34 us; skew 500 ppm
I20260927 02:39:37.585558 9621 webserver.cc:460] Webserver started at http://127.0.98.62:39443/ using document root <none> and password file <none>
I20260927 02:39:37.586108 9621 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:37.586174 9621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:37.586391 9621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:39:37.588367 9621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/master-0/data/instance:
uuid: "74b0684aeff24de189c58477ebbae7d7"
format_stamp: "Formatted at 2026-09-27 02:39:37 on dist-test-slave-zv8z"
server_key: "39c4362eb693e512a97573b84448d750"
server_key_iv: "df706afb38ea3abed5fd05d88259ce74"
server_key_version: "encryptionkey@0"
I20260927 02:39:37.588871 9621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/master-0/wal/instance:
uuid: "74b0684aeff24de189c58477ebbae7d7"
format_stamp: "Formatted at 2026-09-27 02:39:37 on dist-test-slave-zv8z"
server_key: "39c4362eb693e512a97573b84448d750"
server_key_iv: "df706afb38ea3abed5fd05d88259ce74"
server_key_version: "encryptionkey@0"
I20260927 02:39:37.592511 9621 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:39:37.594785 9635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:37.595990 9621 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:39:37.596091 9621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "74b0684aeff24de189c58477ebbae7d7"
format_stamp: "Formatted at 2026-09-27 02:39:37 on dist-test-slave-zv8z"
server_key: "39c4362eb693e512a97573b84448d750"
server_key_iv: "df706afb38ea3abed5fd05d88259ce74"
server_key_version: "encryptionkey@0"
I20260927 02:39:37.596177 9621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:37.611577 9621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:39:37.612205 9621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:39:37.612346 9621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:39:37.618723 9621 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36037
I20260927 02:39:37.618788 9687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36037 every 8 connection(s)
I20260927 02:39:37.619632 9621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:39:37.622642 9688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:39:37.627998 9688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74b0684aeff24de189c58477ebbae7d7: Bootstrap starting.
I20260927 02:39:37.629248 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 9621
I20260927 02:39:37.629344 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/master-0/wal/instance
I20260927 02:39:37.629603 392 external_mini_cluster.cc:1428] Setting key 13ee1c049cb9cf38835f59926e62fd7a
I20260927 02:39:37.630599 9688 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74b0684aeff24de189c58477ebbae7d7: Neither blocks nor log segments found. Creating new log.
I20260927 02:39:37.631445 9688 log.cc:824] T 00000000000000000000000000000000 P 74b0684aeff24de189c58477ebbae7d7: Log is configured to *not* fsync() on all Append() calls
I20260927 02:39:37.633605 9688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74b0684aeff24de189c58477ebbae7d7: No bootstrap required, opened a new log
I20260927 02:39:37.637080 9688 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74b0684aeff24de189c58477ebbae7d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74b0684aeff24de189c58477ebbae7d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36037 } }
I20260927 02:39:37.637291 9688 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74b0684aeff24de189c58477ebbae7d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:39:37.637332 9688 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74b0684aeff24de189c58477ebbae7d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74b0684aeff24de189c58477ebbae7d7, State: Initialized, Role: FOLLOWER
I20260927 02:39:37.637823 9688 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74b0684aeff24de189c58477ebbae7d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74b0684aeff24de189c58477ebbae7d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36037 } }
I20260927 02:39:37.637948 9688 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74b0684aeff24de189c58477ebbae7d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:39:37.638006 9688 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74b0684aeff24de189c58477ebbae7d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:39:37.638098 9688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74b0684aeff24de189c58477ebbae7d7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:39:37.638736 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:37.630665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48696 (local address 127.0.98.62:36037)
0927 02:39:37.631355 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:37.631380 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:37.631427 (+ 47us) server_negotiation.cc:407] Connection header received
0927 02:39:37.632250 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:37.632274 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:37.632662 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:37.632983 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:37.633398 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:37.634434 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:37.635298 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:37.635774 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:37.635920 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:37.636786 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:37.636791 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:37.636803 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:39:37.637193 (+ 390us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:37.637252 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:39:37.637272 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:37.637365 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:37.637535 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 02:39:37.637886 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":148,"threads_started":1}
I20260927 02:39:37.639012 9688 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74b0684aeff24de189c58477ebbae7d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74b0684aeff24de189c58477ebbae7d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36037 } }
I20260927 02:39:37.639321 9688 leader_election.cc:302] T 00000000000000000000000000000000 P 74b0684aeff24de189c58477ebbae7d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74b0684aeff24de189c58477ebbae7d7; no voters:
I20260927 02:39:37.639611 9688 leader_election.cc:288] T 00000000000000000000000000000000 P 74b0684aeff24de189c58477ebbae7d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:39:37.639695 9693 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74b0684aeff24de189c58477ebbae7d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:39:37.639889 9693 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74b0684aeff24de189c58477ebbae7d7 [term 1 LEADER]: Becoming Leader. State: Replica: 74b0684aeff24de189c58477ebbae7d7, State: Running, Role: LEADER
I20260927 02:39:37.640233 9693 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74b0684aeff24de189c58477ebbae7d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74b0684aeff24de189c58477ebbae7d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36037 } }
I20260927 02:39:37.640869 9688 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74b0684aeff24de189c58477ebbae7d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:39:37.641361 9693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74b0684aeff24de189c58477ebbae7d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74b0684aeff24de189c58477ebbae7d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74b0684aeff24de189c58477ebbae7d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36037 } } }
I20260927 02:39:37.641547 9693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74b0684aeff24de189c58477ebbae7d7 [sys.catalog]: This master's current role is: LEADER
I20260927 02:39:37.641690 9694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74b0684aeff24de189c58477ebbae7d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74b0684aeff24de189c58477ebbae7d7. Latest consensus state: current_term: 1 leader_uuid: "74b0684aeff24de189c58477ebbae7d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74b0684aeff24de189c58477ebbae7d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36037 } } }
I20260927 02:39:37.641798 9694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74b0684aeff24de189c58477ebbae7d7 [sys.catalog]: This master's current role is: LEADER
I20260927 02:39:37.642275 9700 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:39:37.645242 9700 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:39:37.650544 9700 catalog_manager.cc:1347] Generated new cluster ID: 716d020ff7ec4538a395ffe2e2f9dd66
I20260927 02:39:37.650612 9700 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:39:37.660265 9700 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:39:37.661366 9700 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:39:37.670343 9700 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74b0684aeff24de189c58477ebbae7d7: Generated new TSK 0
I20260927 02:39:37.670989 9700 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:39:37.675935 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36037
--builtin_ntp_servers=127.0.98.20:38747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 02:39:37.806537 9712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:39:37.806771 9712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:39:37.806821 9712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:39:37.810725 9712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:39:37.810773 9712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:39:37.810845 9712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:39:37.815407 9712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:39:37.816375 9712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:39:37.817662 9712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:39:37.826721 9712 server_base.cc:1034] running on GCE node
W20260927 02:39:37.826690 9718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:37.826686 9717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:37.826668 9720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:37.827280 9712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:39:37.827883 9712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:39:37.829030 9712 hybrid_clock.cc:648] HybridClock initialized: now 1790476777829012 us; error 30 us; skew 500 ppm
I20260927 02:39:37.830760 9712 webserver.cc:460] Webserver started at http://127.0.98.1:39767/ using document root <none> and password file <none>
I20260927 02:39:37.831288 9712 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:37.831348 9712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:37.831550 9712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:39:37.833532 9712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/ts-0/data/instance:
uuid: "e7222dc7a7874d82b6a8a4a53280db1c"
format_stamp: "Formatted at 2026-09-27 02:39:37 on dist-test-slave-zv8z"
server_key: "c52221ccbf5a3bfdccedb5b4edb9d305"
server_key_iv: "56f105a7bd132931c0616c91797a4960"
server_key_version: "encryptionkey@0"
I20260927 02:39:37.834039 9712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e7222dc7a7874d82b6a8a4a53280db1c"
format_stamp: "Formatted at 2026-09-27 02:39:37 on dist-test-slave-zv8z"
server_key: "c52221ccbf5a3bfdccedb5b4edb9d305"
server_key_iv: "56f105a7bd132931c0616c91797a4960"
server_key_version: "encryptionkey@0"
I20260927 02:39:37.837430 9712 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 02:39:37.839785 9726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:37.840984 9712 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 02:39:37.841076 9712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/ts-0/wal
uuid: "e7222dc7a7874d82b6a8a4a53280db1c"
format_stamp: "Formatted at 2026-09-27 02:39:37 on dist-test-slave-zv8z"
server_key: "c52221ccbf5a3bfdccedb5b4edb9d305"
server_key_iv: "56f105a7bd132931c0616c91797a4960"
server_key_version: "encryptionkey@0"
I20260927 02:39:37.841176 9712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:37.854023 9712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:39:37.854676 9712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:39:37.854825 9712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:39:37.855290 9712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:39:37.856191 9712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:39:37.856248 9712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:37.856304 9712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:39:37.856328 9712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:37.864133 9712 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40139
I20260927 02:39:37.864162 9839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40139 every 8 connection(s)
I20260927 02:39:37.864969 9712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:39:37.872177 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 9712
I20260927 02:39:37.872282 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:39:37.872532 392 external_mini_cluster.cc:1428] Setting key ef080be6957011d7e6c79f9ec793f92f
I20260927 02:39:37.873268 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:37.866594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52909 (local address 127.0.98.62:36037)
0927 02:39:37.866736 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:37.866740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:37.867787 (+ 1047us) server_negotiation.cc:407] Connection header received
0927 02:39:37.869042 (+ 1255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:37.869045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:37.869089 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:37.869158 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:37.869924 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:37.870506 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:37.871538 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:37.871742 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:37.871815 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:37.872832 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:37.872836 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:37.872838 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:39:37.872851 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:37.872872 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:39:37.872882 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:37.872941 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:37.873087 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 02:39:37.873141 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 02:39:37.874233 9842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:37.867016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36037 (local address 127.0.98.1:52909)
0927 02:39:37.867624 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:37.867663 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:37.868866 (+ 1203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:37.869285 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:37.869291 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:37.869370 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:37.869825 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:37.869837 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:37.870598 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:37.870601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:37.871449 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:37.871457 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:37.871848 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:37.871850 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:37.871978 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:37.872580 (+ 602us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:39:37.872582 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:37.872632 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:39:37.872708 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:37.872954 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:37.872962 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:37.872967 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:39:37.873081 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 02:39:37.873309 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":100,"threads_started":1}
I20260927 02:39:37.874460 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36037
--builtin_ntp_servers=127.0.98.20:38747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 02:39:37.875329 9840 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36037
I20260927 02:39:37.875545 9840 heartbeater.cc:464] Registering TS with master...
I20260927 02:39:37.876075 9840 heartbeater.cc:511] Master 127.0.98.62:36037 requested a full tablet report, sending...
I20260927 02:39:37.877460 9652 ts_manager.cc:194] Registered new tserver with Master: e7222dc7a7874d82b6a8a4a53280db1c (127.0.98.1:40139)
I20260927 02:39:37.878629 9652 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:52909
W20260927 02:39:37.979391 9843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:39:37.979629 9843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:39:37.979665 9843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:39:37.983698 9843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:39:37.983759 9843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:39:37.983812 9843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:39:37.988428 9843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:39:37.989393 9843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:39:37.990728 9843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:39:37.999786 9849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:37.999830 9851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:37.999830 9848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:38.000180 9843 server_base.cc:1034] running on GCE node
I20260927 02:39:38.000555 9843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:39:38.001173 9843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:39:38.002312 9843 hybrid_clock.cc:648] HybridClock initialized: now 1790476778002283 us; error 40 us; skew 500 ppm
I20260927 02:39:38.004025 9843 webserver.cc:460] Webserver started at http://127.0.98.2:38533/ using document root <none> and password file <none>
I20260927 02:39:38.004603 9843 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:38.004667 9843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:38.004851 9843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:39:38.006886 9843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/ts-1/data/instance:
uuid: "3edc5e075314479594c2b6d95823d91a"
format_stamp: "Formatted at 2026-09-27 02:39:37 on dist-test-slave-zv8z"
server_key: "4dafcbfa4e2ef888276343beed1a3e81"
server_key_iv: "bd5ff4b898d7eeaca0a7152cbcac11b4"
server_key_version: "encryptionkey@0"
I20260927 02:39:38.007416 9843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3edc5e075314479594c2b6d95823d91a"
format_stamp: "Formatted at 2026-09-27 02:39:37 on dist-test-slave-zv8z"
server_key: "4dafcbfa4e2ef888276343beed1a3e81"
server_key_iv: "bd5ff4b898d7eeaca0a7152cbcac11b4"
server_key_version: "encryptionkey@0"
I20260927 02:39:38.010957 9843 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:39:38.013447 9857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:38.014655 9843 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:39:38.014753 9843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/ts-1/wal
uuid: "3edc5e075314479594c2b6d95823d91a"
format_stamp: "Formatted at 2026-09-27 02:39:37 on dist-test-slave-zv8z"
server_key: "4dafcbfa4e2ef888276343beed1a3e81"
server_key_iv: "bd5ff4b898d7eeaca0a7152cbcac11b4"
server_key_version: "encryptionkey@0"
I20260927 02:39:38.014843 9843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:38.033748 9843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:39:38.034432 9843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:39:38.034598 9843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:39:38.035156 9843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:39:38.036073 9843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:39:38.036118 9843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:38.036168 9843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:39:38.036196 9843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:38.044646 9843 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46329
I20260927 02:39:38.044677 9970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46329 every 8 connection(s)
I20260927 02:39:38.045526 9843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:39:38.050323 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 9843
I20260927 02:39:38.050436 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:39:38.050709 392 external_mini_cluster.cc:1428] Setting key 6785e1d06404d2a20d496994c73014ab
I20260927 02:39:38.052692 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36037
--builtin_ntp_servers=127.0.98.20:38747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 02:39:38.055382 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:38.047313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57693 (local address 127.0.98.62:36037)
0927 02:39:38.047434 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:38.047437 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:38.048565 (+ 1128us) server_negotiation.cc:407] Connection header received
0927 02:39:38.049892 (+ 1327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:38.049896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:38.049954 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:38.050029 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:38.051054 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:38.051660 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:38.053293 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:38.053515 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:38.053609 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:38.054821 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:38.054826 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:38.054828 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:39:38.054842 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:38.054866 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:39:38.054876 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:38.054957 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:38.055137 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 02:39:38.055203 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 02:39:38.056301 9973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:38.047720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36037 (local address 127.0.98.2:57693)
0927 02:39:38.048415 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:38.048456 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:38.049682 (+ 1226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:38.050219 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:38.050227 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:38.050353 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:38.050921 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:38.050934 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:38.051786 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:38.051789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:38.053070 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:38.053081 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:38.053660 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:38.053664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:38.053858 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:38.054533 (+ 675us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:39:38.054535 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:38.054588 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:39:38.054657 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:38.054985 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:38.054993 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:38.054998 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:39:38.055125 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 02:39:38.055353 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":114,"threads_started":1}
I20260927 02:39:38.057441 9971 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36037
I20260927 02:39:38.057665 9971 heartbeater.cc:464] Registering TS with master...
I20260927 02:39:38.058177 9971 heartbeater.cc:511] Master 127.0.98.62:36037 requested a full tablet report, sending...
I20260927 02:39:38.059211 9652 ts_manager.cc:194] Registered new tserver with Master: 3edc5e075314479594c2b6d95823d91a (127.0.98.2:46329)
I20260927 02:39:38.059774 9652 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:57693
W20260927 02:39:38.158461 9974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:39:38.158701 9974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:39:38.158748 9974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:39:38.162691 9974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:39:38.162743 9974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:39:38.162801 9974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:39:38.167443 9974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:39:38.168424 9974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:39:38.169737 9974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:39:38.178840 9980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:38.178874 9982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:38.178974 9974 server_base.cc:1034] running on GCE node
W20260927 02:39:38.179147 9979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:38.179550 9974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:39:38.180130 9974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:39:38.181298 9974 hybrid_clock.cc:648] HybridClock initialized: now 1790476778181275 us; error 35 us; skew 500 ppm
I20260927 02:39:38.183205 9974 webserver.cc:460] Webserver started at http://127.0.98.3:42667/ using document root <none> and password file <none>
I20260927 02:39:38.183748 9974 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:38.183812 9974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:38.183987 9974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:39:38.186061 9974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/ts-2/data/instance:
uuid: "cecc915896f64457bf920a4e04ee9c9c"
format_stamp: "Formatted at 2026-09-27 02:39:38 on dist-test-slave-zv8z"
server_key: "cc7f8a20ca7b6b2ab37771ee9005f5d8"
server_key_iv: "eab85d368688dbdb3ff37d19cd0acf58"
server_key_version: "encryptionkey@0"
I20260927 02:39:38.186643 9974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/ts-2/wal/instance:
uuid: "cecc915896f64457bf920a4e04ee9c9c"
format_stamp: "Formatted at 2026-09-27 02:39:38 on dist-test-slave-zv8z"
server_key: "cc7f8a20ca7b6b2ab37771ee9005f5d8"
server_key_iv: "eab85d368688dbdb3ff37d19cd0acf58"
server_key_version: "encryptionkey@0"
I20260927 02:39:38.190356 9974 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:39:38.192828 9988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:38.194028 9974 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:39:38.194140 9974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/ts-2/wal
uuid: "cecc915896f64457bf920a4e04ee9c9c"
format_stamp: "Formatted at 2026-09-27 02:39:38 on dist-test-slave-zv8z"
server_key: "cc7f8a20ca7b6b2ab37771ee9005f5d8"
server_key_iv: "eab85d368688dbdb3ff37d19cd0acf58"
server_key_version: "encryptionkey@0"
I20260927 02:39:38.194244 9974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:38.207154 9974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:39:38.207825 9974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:39:38.207988 9974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:39:38.208561 9974 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:39:38.209450 9974 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:39:38.209496 9974 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:38.209546 9974 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:39:38.209573 9974 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:38.217783 9974 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38223
I20260927 02:39:38.217824 10101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38223 every 8 connection(s)
I20260927 02:39:38.218720 9974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:39:38.227186 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:38.220636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57119 (local address 127.0.98.62:36037)
0927 02:39:38.220777 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:38.220781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:38.221644 (+ 863us) server_negotiation.cc:407] Connection header received
0927 02:39:38.223005 (+ 1361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:38.223009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:38.223054 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:38.223139 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:38.223997 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:38.224513 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:38.225610 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:38.225833 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:38.225905 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:38.226821 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:38.226824 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:38.226826 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:39:38.226837 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:38.226853 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:39:38.226860 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:38.226916 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:38.227028 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 02:39:38.227077 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 02:39:38.228008 10104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:38.220903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36037 (local address 127.0.98.3:57119)
0927 02:39:38.221508 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:38.221546 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:38.222798 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:38.223291 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:38.223297 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:38.223378 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:38.223883 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:38.223896 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:38.224620 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:38.224623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:38.225509 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:38.225518 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:38.225941 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:38.225943 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:38.226076 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:38.226583 (+ 507us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:39:38.226584 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:38.226622 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:39:38.226681 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:38.226920 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:38.226925 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:38.226928 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:39:38.227010 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 02:39:38.227219 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":98,"threads_started":1}
I20260927 02:39:38.228869 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 9974
I20260927 02:39:38.228966 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790476718023131-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:39:38.229004 10102 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36037
I20260927 02:39:38.229209 10102 heartbeater.cc:464] Registering TS with master...
I20260927 02:39:38.229276 392 external_mini_cluster.cc:1428] Setting key e655a00ae0514100995d5bc4ba2fdff2
I20260927 02:39:38.229734 10102 heartbeater.cc:511] Master 127.0.98.62:36037 requested a full tablet report, sending...
I20260927 02:39:38.230767 9651 ts_manager.cc:194] Registered new tserver with Master: cecc915896f64457bf920a4e04ee9c9c (127.0.98.3:38223)
I20260927 02:39:38.231356 9651 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:57119
I20260927 02:39:38.242666 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 02:39:38.245322 9691 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:38.244512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48698 (local address 127.0.98.62:36037)
0927 02:39:38.244625 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:38.244629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:38.244822 (+ 193us) server_negotiation.cc:407] Connection header received
0927 02:39:38.244914 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:38.244916 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:38.244955 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:38.245020 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:38.245178 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48698: BlockingRecv error: recv got EOF from 127.0.0.1:48698 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 02:39:38.245486 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 9712
I20260927 02:39:38.250272 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 9843
I20260927 02:39:38.254746 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 9974
I20260927 02:39:38.258924 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 9621
2026-09-27T02:39:38Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (831 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T02:39:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:39:38Z Disabled control of system clock
I20260927 02:39:38.285123 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43969
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 02:39:38.392778 10113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:39:38.393054 10113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:39:38.393090 10113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:39:38.397047 10113 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:39:38.397119 10113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:39:38.397133 10113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:39:38.397148 10113 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:39:38.397162 10113 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:39:38.401957 10113 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43969
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:39:38.403041 10113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:39:38.404361 10113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:39:38.412475 10119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:38.412470 10118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:38.412545 10113 server_base.cc:1034] running on GCE node
W20260927 02:39:38.412470 10121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:38.413230 10113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:39:38.414412 10113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:39:38.415614 10113 hybrid_clock.cc:648] HybridClock initialized: now 1790476778415577 us; error 30 us; skew 500 ppm
I20260927 02:39:38.417374 10113 webserver.cc:460] Webserver started at http://127.0.98.62:42149/ using document root <none> and password file <none>
I20260927 02:39:38.417937 10113 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:38.417997 10113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:38.418191 10113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:39:38.420190 10113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/master-0/data/instance:
uuid: "7e12e786b9494e69adba5bc50ba9274f"
format_stamp: "Formatted at 2026-09-27 02:39:38 on dist-test-slave-zv8z"
server_key: "9ae41db5cc929791a950f692528df57a"
server_key_iv: "c78ff7319ea69f1c9ed1db1d5023f8cb"
server_key_version: "encryptionkey@0"
I20260927 02:39:38.420730 10113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/master-0/wal/instance:
uuid: "7e12e786b9494e69adba5bc50ba9274f"
format_stamp: "Formatted at 2026-09-27 02:39:38 on dist-test-slave-zv8z"
server_key: "9ae41db5cc929791a950f692528df57a"
server_key_iv: "c78ff7319ea69f1c9ed1db1d5023f8cb"
server_key_version: "encryptionkey@0"
I20260927 02:39:38.424407 10113 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260927 02:39:38.426820 10127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:38.428123 10113 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:39:38.428242 10113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "7e12e786b9494e69adba5bc50ba9274f"
format_stamp: "Formatted at 2026-09-27 02:39:38 on dist-test-slave-zv8z"
server_key: "9ae41db5cc929791a950f692528df57a"
server_key_iv: "c78ff7319ea69f1c9ed1db1d5023f8cb"
server_key_version: "encryptionkey@0"
I20260927 02:39:38.428339 10113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:38.432708 10113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:39:38.433329 10113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:39:38.433461 10113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:39:38.439738 10113 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43969
I20260927 02:39:38.439762 10179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43969 every 8 connection(s)
I20260927 02:39:38.440595 10113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:39:38.443643 10180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:39:38.448827 10180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e12e786b9494e69adba5bc50ba9274f: Bootstrap starting.
I20260927 02:39:38.450083 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 10113
I20260927 02:39:38.450188 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/master-0/wal/instance
I20260927 02:39:38.450510 392 external_mini_cluster.cc:1428] Setting key b0ce379fe6b8bdbb837adcb878a7df50
I20260927 02:39:38.451546 10180 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e12e786b9494e69adba5bc50ba9274f: Neither blocks nor log segments found. Creating new log.
I20260927 02:39:38.452296 10180 log.cc:824] T 00000000000000000000000000000000 P 7e12e786b9494e69adba5bc50ba9274f: Log is configured to *not* fsync() on all Append() calls
I20260927 02:39:38.454412 10180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e12e786b9494e69adba5bc50ba9274f: No bootstrap required, opened a new log
I20260927 02:39:38.455614 10183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:38.451630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42614 (local address 127.0.98.62:43969)
0927 02:39:38.452231 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:38.452246 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:38.452279 (+ 33us) server_negotiation.cc:407] Connection header received
0927 02:39:38.453030 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:38.453054 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:38.453357 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:38.453624 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:38.454377 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:38.454382 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:38.454393 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:39:38.454402 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:38.454454 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:39:38.454469 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:38.454566 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:38.454667 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 02:39:38.454892 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":94,"threads_started":1}
I20260927 02:39:38.458065 10180 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e12e786b9494e69adba5bc50ba9274f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e12e786b9494e69adba5bc50ba9274f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43969 } }
I20260927 02:39:38.458297 10180 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e12e786b9494e69adba5bc50ba9274f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:39:38.458350 10180 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e12e786b9494e69adba5bc50ba9274f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e12e786b9494e69adba5bc50ba9274f, State: Initialized, Role: FOLLOWER
I20260927 02:39:38.458730 10180 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e12e786b9494e69adba5bc50ba9274f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e12e786b9494e69adba5bc50ba9274f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43969 } }
I20260927 02:39:38.458853 10180 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e12e786b9494e69adba5bc50ba9274f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:39:38.458914 10180 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e12e786b9494e69adba5bc50ba9274f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:39:38.459004 10180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e12e786b9494e69adba5bc50ba9274f [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:39:38.459836 10180 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e12e786b9494e69adba5bc50ba9274f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e12e786b9494e69adba5bc50ba9274f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43969 } }
I20260927 02:39:38.460137 10180 leader_election.cc:302] T 00000000000000000000000000000000 P 7e12e786b9494e69adba5bc50ba9274f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e12e786b9494e69adba5bc50ba9274f; no voters:
I20260927 02:39:38.460407 10180 leader_election.cc:288] T 00000000000000000000000000000000 P 7e12e786b9494e69adba5bc50ba9274f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:39:38.460502 10185 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e12e786b9494e69adba5bc50ba9274f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:39:38.460719 10185 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e12e786b9494e69adba5bc50ba9274f [term 1 LEADER]: Becoming Leader. State: Replica: 7e12e786b9494e69adba5bc50ba9274f, State: Running, Role: LEADER
I20260927 02:39:38.461009 10185 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e12e786b9494e69adba5bc50ba9274f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e12e786b9494e69adba5bc50ba9274f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43969 } }
I20260927 02:39:38.461474 10180 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e12e786b9494e69adba5bc50ba9274f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:39:38.462004 10185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e12e786b9494e69adba5bc50ba9274f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e12e786b9494e69adba5bc50ba9274f. Latest consensus state: current_term: 1 leader_uuid: "7e12e786b9494e69adba5bc50ba9274f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e12e786b9494e69adba5bc50ba9274f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43969 } } }
I20260927 02:39:38.462114 10185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e12e786b9494e69adba5bc50ba9274f [sys.catalog]: This master's current role is: LEADER
I20260927 02:39:38.463112 10187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e12e786b9494e69adba5bc50ba9274f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e12e786b9494e69adba5bc50ba9274f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e12e786b9494e69adba5bc50ba9274f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43969 } } }
I20260927 02:39:38.463260 10187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e12e786b9494e69adba5bc50ba9274f [sys.catalog]: This master's current role is: LEADER
W20260927 02:39:38.466099 10200 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7e12e786b9494e69adba5bc50ba9274f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:39:38.466168 10200 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:39:38.466238 10191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:39:38.466910 10191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:39:38.471807 10191 catalog_manager.cc:1347] Generated new cluster ID: 39eee485108d499194bf9ff08ad742a4
I20260927 02:39:38.471905 10191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:39:38.478197 10191 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:39:38.478503 10191 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:39:38.482709 10191 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e12e786b9494e69adba5bc50ba9274f: Generated new TSK 0
I20260927 02:39:38.483300 10191 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:39:38.485683 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43969
--builtin_ntp_servers=127.0.98.20:45989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 02:39:38.595350 10204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:39:38.595604 10204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:39:38.595642 10204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:39:38.599643 10204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:39:38.599699 10204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:39:38.599759 10204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:39:38.604367 10204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:39:38.605319 10204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:39:38.606647 10204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:39:38.615617 10212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:38.615608 10210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:38.615608 10209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:38.615732 10204 server_base.cc:1034] running on GCE node
I20260927 02:39:38.616237 10204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:39:38.616828 10204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:39:38.618014 10204 hybrid_clock.cc:648] HybridClock initialized: now 1790476778617996 us; error 32 us; skew 500 ppm
I20260927 02:39:38.619840 10204 webserver.cc:460] Webserver started at http://127.0.98.1:38953/ using document root <none> and password file <none>
I20260927 02:39:38.620368 10204 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:38.620431 10204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:38.620611 10204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:39:38.622603 10204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/ts-0/data/instance:
uuid: "477f32910d9649cb985bb9ab700ae356"
format_stamp: "Formatted at 2026-09-27 02:39:38 on dist-test-slave-zv8z"
server_key: "932a7bd37186414aa3e7b6213b9c39b6"
server_key_iv: "bbc307a51dd0494177a7998106eee8d9"
server_key_version: "encryptionkey@0"
I20260927 02:39:38.623113 10204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/ts-0/wal/instance:
uuid: "477f32910d9649cb985bb9ab700ae356"
format_stamp: "Formatted at 2026-09-27 02:39:38 on dist-test-slave-zv8z"
server_key: "932a7bd37186414aa3e7b6213b9c39b6"
server_key_iv: "bbc307a51dd0494177a7998106eee8d9"
server_key_version: "encryptionkey@0"
I20260927 02:39:38.626590 10204 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:39:38.628820 10218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:38.629969 10204 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:39:38.630069 10204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/ts-0/wal
uuid: "477f32910d9649cb985bb9ab700ae356"
format_stamp: "Formatted at 2026-09-27 02:39:38 on dist-test-slave-zv8z"
server_key: "932a7bd37186414aa3e7b6213b9c39b6"
server_key_iv: "bbc307a51dd0494177a7998106eee8d9"
server_key_version: "encryptionkey@0"
I20260927 02:39:38.630156 10204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:38.634519 10204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:39:38.635053 10204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:39:38.635200 10204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:39:38.635697 10204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:39:38.636510 10204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:39:38.636554 10204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:38.636602 10204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:39:38.636628 10204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:38.644558 10204 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37719
I20260927 02:39:38.644629 10331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37719 every 8 connection(s)
I20260927 02:39:38.645362 10204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:39:38.646279 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 10204
I20260927 02:39:38.646391 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:39:38.646694 392 external_mini_cluster.cc:1428] Setting key b90051f95bac6b6089cd9c0b11b6139c
I20260927 02:39:38.648562 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43969
--builtin_ntp_servers=127.0.98.20:45989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 02:39:38.651204 10183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:38.647165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42419 (local address 127.0.98.62:43969)
0927 02:39:38.647297 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:38.647300 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:38.648395 (+ 1095us) server_negotiation.cc:407] Connection header received
0927 02:39:38.649648 (+ 1253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:38.649651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:38.649690 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:38.649761 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:38.650793 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:38.650796 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:38.650799 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:39:38.650811 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:38.650833 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:39:38.650841 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:38.650918 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:38.651038 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 02:39:38.651078 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 02:39:38.652043 10334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:38.647571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43969 (local address 127.0.98.1:42419)
0927 02:39:38.648250 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:38.648305 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:38.649460 (+ 1155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:38.649924 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:38.649930 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:38.650016 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:38.650561 (+ 545us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:39:38.650562 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:38.650597 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:39:38.650659 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:38.650910 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:38.650917 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:38.650923 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 02:39:38.651068 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 02:39:38.651296 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":138,"threads_started":1}
I20260927 02:39:38.653223 10332 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43969
I20260927 02:39:38.653452 10332 heartbeater.cc:464] Registering TS with master...
I20260927 02:39:38.653824 10332 heartbeater.cc:511] Master 127.0.98.62:43969 requested a full tablet report, sending...
I20260927 02:39:38.655385 10144 ts_manager.cc:194] Registered new tserver with Master: 477f32910d9649cb985bb9ab700ae356 (127.0.98.1:37719)
W20260927 02:39:38.753589 10335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:39:38.753844 10335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:39:38.753881 10335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:39:38.758188 10335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:39:38.758242 10335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:39:38.758332 10335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:39:38.763010 10335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:39:38.763985 10335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:39:38.765252 10335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:39:38.774385 10341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:38.774382 10340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:38.774431 10335 server_base.cc:1034] running on GCE node
W20260927 02:39:38.774389 10343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:38.774947 10335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:39:38.775540 10335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:39:38.776719 10335 hybrid_clock.cc:648] HybridClock initialized: now 1790476778776697 us; error 32 us; skew 500 ppm
I20260927 02:39:38.778611 10335 webserver.cc:460] Webserver started at http://127.0.98.2:33173/ using document root <none> and password file <none>
I20260927 02:39:38.779170 10335 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:38.779237 10335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:38.779417 10335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:39:38.781381 10335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/ts-1/data/instance:
uuid: "e90380ae7bc24941a970411eb226078d"
format_stamp: "Formatted at 2026-09-27 02:39:38 on dist-test-slave-zv8z"
server_key: "2325cd09b6c6c754cec40251a66ab93f"
server_key_iv: "f36633bb06233efeb87ca4a897153e9d"
server_key_version: "encryptionkey@0"
I20260927 02:39:38.781906 10335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e90380ae7bc24941a970411eb226078d"
format_stamp: "Formatted at 2026-09-27 02:39:38 on dist-test-slave-zv8z"
server_key: "2325cd09b6c6c754cec40251a66ab93f"
server_key_iv: "f36633bb06233efeb87ca4a897153e9d"
server_key_version: "encryptionkey@0"
I20260927 02:39:38.785427 10335 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 02:39:38.787700 10349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:38.788870 10335 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:39:38.788969 10335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/ts-1/wal
uuid: "e90380ae7bc24941a970411eb226078d"
format_stamp: "Formatted at 2026-09-27 02:39:38 on dist-test-slave-zv8z"
server_key: "2325cd09b6c6c754cec40251a66ab93f"
server_key_iv: "f36633bb06233efeb87ca4a897153e9d"
server_key_version: "encryptionkey@0"
I20260927 02:39:38.789057 10335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:38.793387 10335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:39:38.793937 10335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:39:38.794085 10335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:39:38.794607 10335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:39:38.795430 10335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:39:38.795476 10335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:38.795524 10335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:39:38.795552 10335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:38.803720 10335 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42979
I20260927 02:39:38.803778 10462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42979 every 8 connection(s)
I20260927 02:39:38.804575 10335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:39:38.810065 10183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:38.806151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47621 (local address 127.0.98.62:43969)
0927 02:39:38.806334 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:38.806338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:38.807358 (+ 1020us) server_negotiation.cc:407] Connection header received
0927 02:39:38.808577 (+ 1219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:38.808579 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:38.808614 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:38.808687 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:38.809718 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:38.809721 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:38.809723 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:39:38.809734 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:38.809770 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:39:38.809778 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:38.809848 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:38.809930 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 02:39:38.809962 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 02:39:38.810882 10465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:38.806605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43969 (local address 127.0.98.2:47621)
0927 02:39:38.807222 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:38.807259 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:38.808391 (+ 1132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:38.808847 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:38.808853 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:38.808938 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:38.809477 (+ 539us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:39:38.809479 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:38.809519 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:39:38.809583 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:38.809836 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:38.809840 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:38.809843 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:39:38.809962 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 02:39:38.810151 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":111,"threads_started":1}
I20260927 02:39:38.811831 10463 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43969
I20260927 02:39:38.812039 10463 heartbeater.cc:464] Registering TS with master...
I20260927 02:39:38.812393 10463 heartbeater.cc:511] Master 127.0.98.62:43969 requested a full tablet report, sending...
I20260927 02:39:38.813316 10144 ts_manager.cc:194] Registered new tserver with Master: e90380ae7bc24941a970411eb226078d (127.0.98.2:42979)
I20260927 02:39:38.814010 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 10335
I20260927 02:39:38.814107 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:39:38.814407 392 external_mini_cluster.cc:1428] Setting key 090fe7239ceced7ee4ee287b8c409315
I20260927 02:39:38.816035 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43969
--builtin_ntp_servers=127.0.98.20:45989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 02:39:38.920539 10466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:39:38.920773 10466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:39:38.920807 10466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:39:38.924707 10466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:39:38.924752 10466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:39:38.924808 10466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:39:38.929388 10466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:39:38.930343 10466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:39:38.931602 10466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:39:38.940515 10471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:38.940560 10466 server_base.cc:1034] running on GCE node
W20260927 02:39:38.940593 10474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:38.940692 10472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:38.941257 10466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:39:38.941838 10466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:39:38.943001 10466 hybrid_clock.cc:648] HybridClock initialized: now 1790476778942973 us; error 39 us; skew 500 ppm
I20260927 02:39:38.944746 10466 webserver.cc:460] Webserver started at http://127.0.98.3:35693/ using document root <none> and password file <none>
I20260927 02:39:38.945284 10466 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:38.945348 10466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:38.945528 10466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:39:38.947489 10466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/ts-2/data/instance:
uuid: "d83a2241aeee4a648b69c7f5ca1339ab"
format_stamp: "Formatted at 2026-09-27 02:39:38 on dist-test-slave-zv8z"
server_key: "8800fab86f9e2291fe49a68f7813d37a"
server_key_iv: "02e2dd232fbc27319a8e660a75db3437"
server_key_version: "encryptionkey@0"
I20260927 02:39:38.947984 10466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d83a2241aeee4a648b69c7f5ca1339ab"
format_stamp: "Formatted at 2026-09-27 02:39:38 on dist-test-slave-zv8z"
server_key: "8800fab86f9e2291fe49a68f7813d37a"
server_key_iv: "02e2dd232fbc27319a8e660a75db3437"
server_key_version: "encryptionkey@0"
I20260927 02:39:38.951393 10466 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:39:38.953671 10480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:38.954839 10466 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:39:38.954952 10466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/ts-2/wal
uuid: "d83a2241aeee4a648b69c7f5ca1339ab"
format_stamp: "Formatted at 2026-09-27 02:39:38 on dist-test-slave-zv8z"
server_key: "8800fab86f9e2291fe49a68f7813d37a"
server_key_iv: "02e2dd232fbc27319a8e660a75db3437"
server_key_version: "encryptionkey@0"
I20260927 02:39:38.955060 10466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:38.959216 10466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:39:38.959755 10466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:39:38.959877 10466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:39:38.960318 10466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:39:38.961095 10466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:39:38.961145 10466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:38.961179 10466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:39:38.961192 10466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:38.969076 10466 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44929
I20260927 02:39:38.969099 10593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44929 every 8 connection(s)
I20260927 02:39:38.969864 10466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:39:38.971276 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 10466
I20260927 02:39:38.971366 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790476718023131-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:39:38.971642 392 external_mini_cluster.cc:1428] Setting key a22ad09245b408bbd4638ca55239f950
I20260927 02:39:38.975710 10183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:38.971686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45357 (local address 127.0.98.62:43969)
0927 02:39:38.971826 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:38.971830 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:38.973011 (+ 1181us) server_negotiation.cc:407] Connection header received
0927 02:39:38.974350 (+ 1339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:38.974353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:38.974387 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:38.974451 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:38.975406 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:38.975408 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:38.975410 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:39:38.975417 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:38.975432 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:39:38.975439 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:38.975497 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:38.975584 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 02:39:38.975615 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 02:39:38.976516 10596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:38.972129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43969 (local address 127.0.98.3:45357)
0927 02:39:38.972876 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:38.972928 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:38.974156 (+ 1228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:38.974577 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:38.974584 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:38.974673 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:38.975192 (+ 519us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:39:38.975194 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:38.975231 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:39:38.975297 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:38.975488 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:38.975492 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:38.975496 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:39:38.975611 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 02:39:38.975810 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":142,"threads_started":1}
I20260927 02:39:38.977540 10594 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43969
I20260927 02:39:38.977748 10594 heartbeater.cc:464] Registering TS with master...
I20260927 02:39:38.978101 10594 heartbeater.cc:511] Master 127.0.98.62:43969 requested a full tablet report, sending...
I20260927 02:39:38.979035 10144 ts_manager.cc:194] Registered new tserver with Master: d83a2241aeee4a648b69c7f5ca1339ab (127.0.98.3:44929)
I20260927 02:39:38.985153 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 02:39:38.987686 10183 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:39:38.986979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42628 (local address 127.0.98.62:43969)
0927 02:39:38.987086 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:38.987090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:38.987225 (+ 135us) server_negotiation.cc:407] Connection header received
0927 02:39:38.987327 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:38.987329 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:38.987359 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:38.987422 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:38.987580 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42628: BlockingRecv error: recv got EOF from 127.0.0.1:42628 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 02:39:38.987886 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 10204
I20260927 02:39:38.992359 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 10335
I20260927 02:39:38.996587 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 10466
I20260927 02:39:39.000926 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 10113
2026-09-27T02:39:38Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (741 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:39:39 dist-test-slave-zv8z krb5kdc[10604](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:39:39 dist-test-slave-zv8z krb5kdc[10604](info): set up 2 sockets
Sep 27 02:39:39 dist-test-slave-zv8z krb5kdc[10604](info): commencing operation
krb5kdc: starting...
W20260927 02:39:39.954897 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/krb5kdc/test-user.keytab.
Sep 27 02:39:40 dist-test-slave-zv8z krb5kdc[10604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:39:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:39:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:39:40.069460 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39563
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39563
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:39:40.174677 10617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:39:40.175033 10617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:39:40.175104 10617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:39:40.179051 10617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:39:40.179106 10617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:39:40.179133 10617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:39:40.179163 10617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:39:40.179193 10617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:39:40.184710 10617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39563
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:39:40.185729 10617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:39:40.186750 10617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:39:40.194798 10623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:40.194854 10617 server_base.cc:1034] running on GCE node
W20260927 02:39:40.194782 10625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:40.194790 10622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:40.195556 10617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:39:40.196637 10617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:39:40.197809 10617 hybrid_clock.cc:648] HybridClock initialized: now 1790476780197787 us; error 30 us; skew 500 ppm
Sep 27 02:39:40 dist-test-slave-zv8z krb5kdc[10604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476780, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:39:40.203424 10617 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:39:40.204319 10617 webserver.cc:460] Webserver started at http://127.0.98.62:36765/ using document root <none> and password file <none>
I20260927 02:39:40.204856 10617 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:40.204921 10617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:40.205102 10617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:39:40.207147 10617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/master-0/data/instance:
uuid: "274fca3ba5704aa49474b3fe1248af72"
format_stamp: "Formatted at 2026-09-27 02:39:40 on dist-test-slave-zv8z"
server_key: "55cf5b0ac8a2db9ff754ae5c4a8c53b5"
server_key_iv: "5af6268624d688716939b8ed8608c822"
server_key_version: "encryptionkey@0"
I20260927 02:39:40.207676 10617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/master-0/wal/instance:
uuid: "274fca3ba5704aa49474b3fe1248af72"
format_stamp: "Formatted at 2026-09-27 02:39:40 on dist-test-slave-zv8z"
server_key: "55cf5b0ac8a2db9ff754ae5c4a8c53b5"
server_key_iv: "5af6268624d688716939b8ed8608c822"
server_key_version: "encryptionkey@0"
I20260927 02:39:40.211149 10617 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 02:39:40.213481 10632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:40.214721 10617 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:39:40.214828 10617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "274fca3ba5704aa49474b3fe1248af72"
format_stamp: "Formatted at 2026-09-27 02:39:40 on dist-test-slave-zv8z"
server_key: "55cf5b0ac8a2db9ff754ae5c4a8c53b5"
server_key_iv: "5af6268624d688716939b8ed8608c822"
server_key_version: "encryptionkey@0"
I20260927 02:39:40.214913 10617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:40.235663 10617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:39:40.239327 10617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:39:40.239512 10617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:39:40.246541 10617 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39563
I20260927 02:39:40.246878 10684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39563 every 8 connection(s)
I20260927 02:39:40.247501 10617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:39:40.250798 10685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:39:40.255108 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 10617
I20260927 02:39:40.255275 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/master-0/wal/instance
I20260927 02:39:40.255543 392 external_mini_cluster.cc:1428] Setting key 7fe57120e288f1b5dd7e847660a6799f
I20260927 02:39:40.257366 10685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 274fca3ba5704aa49474b3fe1248af72: Bootstrap starting.
I20260927 02:39:40.259563 10685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 274fca3ba5704aa49474b3fe1248af72: Neither blocks nor log segments found. Creating new log.
I20260927 02:39:40.260445 10685 log.cc:824] T 00000000000000000000000000000000 P 274fca3ba5704aa49474b3fe1248af72: Log is configured to *not* fsync() on all Append() calls
I20260927 02:39:40.262173 10685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 274fca3ba5704aa49474b3fe1248af72: No bootstrap required, opened a new log
Sep 27 02:39:40 dist-test-slave-zv8z krb5kdc[10604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:39:40.266161 10685 raft_consensus.cc:348] T 00000000000000000000000000000000 P 274fca3ba5704aa49474b3fe1248af72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "274fca3ba5704aa49474b3fe1248af72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39563 } }
I20260927 02:39:40.266424 10685 raft_consensus.cc:374] T 00000000000000000000000000000000 P 274fca3ba5704aa49474b3fe1248af72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:39:40.266505 10685 raft_consensus.cc:729] T 00000000000000000000000000000000 P 274fca3ba5704aa49474b3fe1248af72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 274fca3ba5704aa49474b3fe1248af72, State: Initialized, Role: FOLLOWER
I20260927 02:39:40.266953 10685 consensus_queue.cc:260] T 00000000000000000000000000000000 P 274fca3ba5704aa49474b3fe1248af72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "274fca3ba5704aa49474b3fe1248af72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39563 } }
I20260927 02:39:40.267118 10685 raft_consensus.cc:388] T 00000000000000000000000000000000 P 274fca3ba5704aa49474b3fe1248af72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:39:40.267210 10685 raft_consensus.cc:482] T 00000000000000000000000000000000 P 274fca3ba5704aa49474b3fe1248af72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:39:40.267303 10685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 274fca3ba5704aa49474b3fe1248af72 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:39:40.268270 10685 raft_consensus.cc:504] T 00000000000000000000000000000000 P 274fca3ba5704aa49474b3fe1248af72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "274fca3ba5704aa49474b3fe1248af72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39563 } }
I20260927 02:39:40.268610 10685 leader_election.cc:302] T 00000000000000000000000000000000 P 274fca3ba5704aa49474b3fe1248af72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 274fca3ba5704aa49474b3fe1248af72; no voters:
I20260927 02:39:40.272850 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:40.257333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59930 (local address 127.0.98.62:39563)
0927 02:39:40.257774 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:40.257794 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:40.257839 (+ 45us) server_negotiation.cc:407] Connection header received
0927 02:39:40.258744 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:40.258769 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:40.259221 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:40.259597 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:40.260392 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:40.261916 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:40.263089 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:40.263815 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:40.263971 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:40.266862 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:40.266868 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:40.266883 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:40.266911 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:40.269395 (+ 2484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:40.269864 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:40.269869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:40.269875 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:40.270001 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:40.270310 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:40.270313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:40.270315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:40.270767 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:40.271528 (+ 761us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:40.271671 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 02:39:40.271942 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":123,"threads_started":1}
I20260927 02:39:40.274737 10685 leader_election.cc:288] T 00000000000000000000000000000000 P 274fca3ba5704aa49474b3fe1248af72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:39:40.274802 10690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 274fca3ba5704aa49474b3fe1248af72 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:39:40.275045 10690 raft_consensus.cc:686] T 00000000000000000000000000000000 P 274fca3ba5704aa49474b3fe1248af72 [term 1 LEADER]: Becoming Leader. State: Replica: 274fca3ba5704aa49474b3fe1248af72, State: Running, Role: LEADER
I20260927 02:39:40.275373 10690 consensus_queue.cc:237] T 00000000000000000000000000000000 P 274fca3ba5704aa49474b3fe1248af72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "274fca3ba5704aa49474b3fe1248af72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39563 } }
I20260927 02:39:40.276152 10685 sys_catalog.cc:564] T 00000000000000000000000000000000 P 274fca3ba5704aa49474b3fe1248af72 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:39:40.276950 10691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 274fca3ba5704aa49474b3fe1248af72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 274fca3ba5704aa49474b3fe1248af72. Latest consensus state: current_term: 1 leader_uuid: "274fca3ba5704aa49474b3fe1248af72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "274fca3ba5704aa49474b3fe1248af72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39563 } } }
I20260927 02:39:40.277078 10691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 274fca3ba5704aa49474b3fe1248af72 [sys.catalog]: This master's current role is: LEADER
I20260927 02:39:40.277282 10691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 274fca3ba5704aa49474b3fe1248af72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "274fca3ba5704aa49474b3fe1248af72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "274fca3ba5704aa49474b3fe1248af72" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39563 } } }
I20260927 02:39:40.277375 10691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 274fca3ba5704aa49474b3fe1248af72 [sys.catalog]: This master's current role is: LEADER
W20260927 02:39:40.281652 10705 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 274fca3ba5704aa49474b3fe1248af72: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:39:40.281730 10705 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:39:40.281785 10697 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:39:40.282572 10697 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:39:40.287860 10697 catalog_manager.cc:1347] Generated new cluster ID: ed672ab82bc24b709d5b45824e1633cd
I20260927 02:39:40.287927 10697 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:39:40.319491 10697 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:39:40.320505 10697 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:39:40.324810 10697 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 274fca3ba5704aa49474b3fe1248af72: Generated new TSK 0
I20260927 02:39:40.325429 10697 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:39:40.358508 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39563
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:39:40.488555 10711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:39:40.488807 10711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:39:40.488848 10711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:39:40.492733 10711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:39:40.492785 10711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:39:40.492847 10711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:39:40.498005 10711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:39:40.499030 10711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:39:40.499997 10711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:39:40.509275 10717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:40.509306 10719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:40.509275 10716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:40.509306 10711 server_base.cc:1034] running on GCE node
I20260927 02:39:40.509888 10711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:39:40.510558 10711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:39:40.511718 10711 hybrid_clock.cc:648] HybridClock initialized: now 1790476780511701 us; error 30 us; skew 500 ppm
Sep 27 02:39:40 dist-test-slave-zv8z krb5kdc[10604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476780, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:39:40.514420 10711 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:39:40.515262 10711 webserver.cc:460] Webserver started at http://127.0.98.1:37973/ using document root <none> and password file <none>
I20260927 02:39:40.515789 10711 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:40.515861 10711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:40.516068 10711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:39:40.518033 10711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/ts-0/data/instance:
uuid: "78f99620825f4c73b79001e1d5c48053"
format_stamp: "Formatted at 2026-09-27 02:39:40 on dist-test-slave-zv8z"
server_key: "463027af2b43d72e48b3ceed506b33c7"
server_key_iv: "779283510481332a4d28ca0babe50c55"
server_key_version: "encryptionkey@0"
I20260927 02:39:40.518586 10711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/ts-0/wal/instance:
uuid: "78f99620825f4c73b79001e1d5c48053"
format_stamp: "Formatted at 2026-09-27 02:39:40 on dist-test-slave-zv8z"
server_key: "463027af2b43d72e48b3ceed506b33c7"
server_key_iv: "779283510481332a4d28ca0babe50c55"
server_key_version: "encryptionkey@0"
I20260927 02:39:40.522006 10711 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:39:40.524256 10726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:40.525415 10711 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:39:40.525516 10711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/ts-0/wal
uuid: "78f99620825f4c73b79001e1d5c48053"
format_stamp: "Formatted at 2026-09-27 02:39:40 on dist-test-slave-zv8z"
server_key: "463027af2b43d72e48b3ceed506b33c7"
server_key_iv: "779283510481332a4d28ca0babe50c55"
server_key_version: "encryptionkey@0"
I20260927 02:39:40.525610 10711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:40.537415 10711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:39:40.540091 10711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:39:40.540247 10711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:39:40.540772 10711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:39:40.541626 10711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:39:40.541671 10711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:40.541709 10711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:39:40.541736 10711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:40.550635 10711 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44833
I20260927 02:39:40.550657 10839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44833 every 8 connection(s)
I20260927 02:39:40.551532 10711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:39:40.554713 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 10711
I20260927 02:39:40.554803 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:39:40.555058 392 external_mini_cluster.cc:1428] Setting key 6c1a0d850169fd046299e4c77a4119ed
Sep 27 02:39:40 dist-test-slave-zv8z krb5kdc[10604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476780, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:39:40.565367 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:40.553408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56699 (local address 127.0.98.62:39563)
0927 02:39:40.553558 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:40.553561 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:40.554404 (+ 843us) server_negotiation.cc:407] Connection header received
0927 02:39:40.555752 (+ 1348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:40.555755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:40.555816 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:40.555886 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:40.557156 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:40.557757 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:40.559006 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:40.559336 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:40.559438 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:40.562136 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:40.562139 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:40.562141 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:40.562163 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:40.563471 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:40.564043 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:40.564046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:40.564048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:40.564120 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:40.564418 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:40.564420 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:40.564421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:40.564576 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:40.564642 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:40.565154 (+ 512us) server_negotiation.cc:299] Negotiation successful
0927 02:39:40.565248 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 02:39:40.566176 10842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:40.553754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.1:56699)
0927 02:39:40.554210 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:40.554277 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:40.555557 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:40.556063 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:40.556070 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:40.556517 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:40.557017 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:40.557030 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:40.557922 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:40.557925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:40.558873 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:40.558883 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:40.559461 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:40.559463 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:40.559618 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:40.560200 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:40.560222 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:40.561955 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:40.563588 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:40.563594 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:40.563610 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:40.563936 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:40.564202 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:40.564205 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:40.564207 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:40.564350 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:40.564646 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:40.564651 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:40.564941 (+ 290us) client_negotiation.cc:769] Sending connection context
0927 02:39:40.565104 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 02:39:40.565344 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":132,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 02:39:40.567337 10840 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39563
I20260927 02:39:40.567575 10840 heartbeater.cc:464] Registering TS with master...
I20260927 02:39:40.568022 10840 heartbeater.cc:511] Master 127.0.98.62:39563 requested a full tablet report, sending...
I20260927 02:39:40.569509 10649 ts_manager.cc:194] Registered new tserver with Master: 78f99620825f4c73b79001e1d5c48053 (127.0.98.1:44833)
I20260927 02:39:40.571102 10649 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56699
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:39:40.581034 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39563
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:39:40.686358 10845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:39:40.686617 10845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:39:40.686658 10845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:39:40.690615 10845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:39:40.690667 10845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:39:40.690729 10845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:39:40.695802 10845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:39:40.696739 10845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:39:40.697707 10845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:39:40.706730 10853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:40.706712 10850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:40.706818 10851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:40.707190 10845 server_base.cc:1034] running on GCE node
I20260927 02:39:40.707536 10845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:39:40.708117 10845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:39:40.709278 10845 hybrid_clock.cc:648] HybridClock initialized: now 1790476780709246 us; error 48 us; skew 500 ppm
Sep 27 02:39:40 dist-test-slave-zv8z krb5kdc[10604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476780, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:39:40.712010 10845 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:39:40.712816 10845 webserver.cc:460] Webserver started at http://127.0.98.2:39771/ using document root <none> and password file <none>
I20260927 02:39:40.713335 10845 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:40.713397 10845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:40.713578 10845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:39:40.715553 10845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/ts-1/data/instance:
uuid: "f82c119fe4284a839dd48d9795d642ec"
format_stamp: "Formatted at 2026-09-27 02:39:40 on dist-test-slave-zv8z"
server_key: "a5641aa6dc15aa06328a7a7c359f4a69"
server_key_iv: "73687e91d9af1969478d2c344979279c"
server_key_version: "encryptionkey@0"
I20260927 02:39:40.716081 10845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f82c119fe4284a839dd48d9795d642ec"
format_stamp: "Formatted at 2026-09-27 02:39:40 on dist-test-slave-zv8z"
server_key: "a5641aa6dc15aa06328a7a7c359f4a69"
server_key_iv: "73687e91d9af1969478d2c344979279c"
server_key_version: "encryptionkey@0"
I20260927 02:39:40.719544 10845 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:39:40.721729 10860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:40.722926 10845 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:39:40.723026 10845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/ts-1/wal
uuid: "f82c119fe4284a839dd48d9795d642ec"
format_stamp: "Formatted at 2026-09-27 02:39:40 on dist-test-slave-zv8z"
server_key: "a5641aa6dc15aa06328a7a7c359f4a69"
server_key_iv: "73687e91d9af1969478d2c344979279c"
server_key_version: "encryptionkey@0"
I20260927 02:39:40.723111 10845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:40.736001 10845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:39:40.738708 10845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:39:40.738864 10845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:39:40.739379 10845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:39:40.740119 10845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:39:40.740165 10845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:40.740213 10845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:39:40.740242 10845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:40.747978 10845 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44811
I20260927 02:39:40.748003 10973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44811 every 8 connection(s)
I20260927 02:39:40.748798 10845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:39:40.756628 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 10845
I20260927 02:39:40.756739 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:39:40.757057 392 external_mini_cluster.cc:1428] Setting key 8f4e308cf63f802c18a050561fb56043
Sep 27 02:39:40 dist-test-slave-zv8z krb5kdc[10604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476780, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:39:40.762043 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:40.750570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49739 (local address 127.0.98.62:39563)
0927 02:39:40.750685 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:40.750689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:40.751615 (+ 926us) server_negotiation.cc:407] Connection header received
0927 02:39:40.752845 (+ 1230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:40.752848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:40.752898 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:40.752971 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:40.754060 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:40.754653 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:40.755746 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:40.755966 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:40.756045 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:40.758491 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:40.758494 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:40.758497 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:40.758524 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:40.760031 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:40.760627 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:40.760630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:40.760631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:40.760704 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:40.761037 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:40.761039 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:40.761040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:40.761234 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:40.761308 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:40.761835 (+ 527us) server_negotiation.cc:299] Negotiation successful
0927 02:39:40.761929 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 02:39:40.763195 10976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:40.750976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.2:49739)
0927 02:39:40.751462 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:40.751519 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:40.752631 (+ 1112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:40.753096 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:40.753102 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:40.753427 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:40.753909 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:40.753922 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:40.754754 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:40.754757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:40.755638 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:40.755646 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:40.756080 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:40.756082 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:40.756213 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:40.756705 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:40.756726 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:40.758332 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:40.760158 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:40.760164 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:40.760179 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:40.760536 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:40.760780 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:40.760783 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:40.760785 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:40.760955 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:40.761328 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:40.761335 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:40.761618 (+ 283us) client_negotiation.cc:769] Sending connection context
0927 02:39:40.761812 (+ 194us) client_negotiation.cc:240] Negotiation successful
0927 02:39:40.762121 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":135,"threads_started":1}
I20260927 02:39:40.764310 10974 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39563
I20260927 02:39:40.764529 10974 heartbeater.cc:464] Registering TS with master...
I20260927 02:39:40.764964 10974 heartbeater.cc:511] Master 127.0.98.62:39563 requested a full tablet report, sending...
I20260927 02:39:40.765928 10649 ts_manager.cc:194] Registered new tserver with Master: f82c119fe4284a839dd48d9795d642ec (127.0.98.2:44811)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:39:40.766517 10649 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49739
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:39:40.783131 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39563
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:39:40.889290 10979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:39:40.889532 10979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:39:40.889570 10979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:39:40.893499 10979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:39:40.893560 10979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:39:40.893615 10979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:39:40.898854 10979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:39:40.899803 10979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:39:40.900722 10979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:39:40.910034 10987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:40.910105 10979 server_base.cc:1034] running on GCE node
W20260927 02:39:40.910034 10985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:39:40.910028 10984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:39:40.910605 10979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:39:40.911175 10979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:39:40.912325 10979 hybrid_clock.cc:648] HybridClock initialized: now 1790476780912303 us; error 36 us; skew 500 ppm
Sep 27 02:39:40 dist-test-slave-zv8z krb5kdc[10604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476780, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:39:40.914847 10979 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:39:40.915642 10979 webserver.cc:460] Webserver started at http://127.0.98.3:37785/ using document root <none> and password file <none>
I20260927 02:39:40.916147 10979 fs_manager.cc:359] Metadata directory not provided
I20260927 02:39:40.916203 10979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:39:40.916383 10979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:39:40.918346 10979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/ts-2/data/instance:
uuid: "e93475820afa458d9b84e82aa55cd089"
format_stamp: "Formatted at 2026-09-27 02:39:40 on dist-test-slave-zv8z"
server_key: "f1d6f896cdb801b7d1c942444bf9c72f"
server_key_iv: "93666410ee6d67e04561e0bc8082417a"
server_key_version: "encryptionkey@0"
I20260927 02:39:40.918859 10979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e93475820afa458d9b84e82aa55cd089"
format_stamp: "Formatted at 2026-09-27 02:39:40 on dist-test-slave-zv8z"
server_key: "f1d6f896cdb801b7d1c942444bf9c72f"
server_key_iv: "93666410ee6d67e04561e0bc8082417a"
server_key_version: "encryptionkey@0"
I20260927 02:39:40.922291 10979 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:39:40.924445 10994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:40.925581 10979 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:39:40.925674 10979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/ts-2/wal
uuid: "e93475820afa458d9b84e82aa55cd089"
format_stamp: "Formatted at 2026-09-27 02:39:40 on dist-test-slave-zv8z"
server_key: "f1d6f896cdb801b7d1c942444bf9c72f"
server_key_iv: "93666410ee6d67e04561e0bc8082417a"
server_key_version: "encryptionkey@0"
I20260927 02:39:40.925767 10979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:39:40.944128 10979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:39:40.947263 10979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:39:40.947428 10979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:39:40.947988 10979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:39:40.948768 10979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:39:40.948812 10979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:40.948850 10979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:39:40.948879 10979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:39:40.957007 10979 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44245
I20260927 02:39:40.957037 11107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44245 every 8 connection(s)
I20260927 02:39:40.957849 10979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:39:40.958633 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 10979
I20260927 02:39:40.958742 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:39:40.958999 392 external_mini_cluster.cc:1428] Setting key dbfcd2bce7922b9dfbe3686e61d3ed05
Sep 27 02:39:40 dist-test-slave-zv8z krb5kdc[10604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476780, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:39:40.969775 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:40.960888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49213 (local address 127.0.98.62:39563)
0927 02:39:40.961003 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:40.961008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:40.961023 (+ 15us) server_negotiation.cc:407] Connection header received
0927 02:39:40.961767 (+ 744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:40.961770 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:40.961819 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:40.961888 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:40.963022 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:40.963533 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:40.964610 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:40.964820 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:40.964911 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:40.967118 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:40.967121 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:40.967122 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:40.967143 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:40.968251 (+ 1108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:40.968652 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:40.968654 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:40.968656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:40.968722 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:40.968990 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:40.968992 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:40.968993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:40.969140 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:40.969205 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:40.969580 (+ 375us) server_negotiation.cc:299] Negotiation successful
0927 02:39:40.969661 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 02:39:40.970556 11110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:40.959967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39563 (local address 127.0.98.3:49213)
0927 02:39:40.960408 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:40.960449 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:40.961587 (+ 1138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:40.962049 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:40.962056 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:40.962432 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:39:40.962898 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:40.962911 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:40.963639 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:40.963642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:40.964502 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:40.964511 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:40.964943 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:40.964946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:40.965104 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:40.965593 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:39:40.965607 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:39:40.966987 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:39:40.968339 (+ 1352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:40.968344 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:40.968354 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:40.968578 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:40.968810 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:39:40.968812 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:39:40.968814 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:39:40.968928 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:39:40.969205 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:39:40.969209 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:39:40.969414 (+ 205us) client_negotiation.cc:769] Sending connection context
0927 02:39:40.969550 (+ 136us) client_negotiation.cc:240] Negotiation successful
0927 02:39:40.969795 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":111,"threads_started":1}
I20260927 02:39:40.971611 11108 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39563
I20260927 02:39:40.972011 11108 heartbeater.cc:464] Registering TS with master...
I20260927 02:39:40.972497 11108 heartbeater.cc:511] Master 127.0.98.62:39563 requested a full tablet report, sending...
I20260927 02:39:40.973376 10649 ts_manager.cc:194] Registered new tserver with Master: e93475820afa458d9b84e82aa55cd089 (127.0.98.3:44245)
I20260927 02:39:40.973886 10649 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49213
I20260927 02:39:40.983922 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:39:40.991928 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:40.985658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38196 (local address 127.0.98.62:39563)
0927 02:39:40.985760 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:40.985763 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:40.985902 (+ 139us) server_negotiation.cc:407] Connection header received
0927 02:39:40.985998 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:40.986000 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:40.986033 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:40.986092 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:39:40.986734 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:40.987220 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:40.987990 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:40.988184 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:40.988267 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:40.989426 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:39:40.989429 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:39:40.989431 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:39:40.989447 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:39:40.990562 (+ 1115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:40.990905 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:40.990907 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:40.990908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:40.990972 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:39:40.991257 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:39:40.991259 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:39:40.991261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:39:40.991441 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:39:40.991531 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:40.991731 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 02:39:40.991811 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 02:39:40.994508 10649 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38196:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:39:40.997323 10649 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:39:41.010219 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:41.004723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44811 (local address 127.0.0.1:46038)
0927 02:39:41.005147 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:41.005197 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:41.005316 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:41.006061 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:41.006067 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:41.006085 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:39:41.006203 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:41.006209 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:41.007551 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:41.007554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:41.008677 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:41.008688 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:41.009705 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:41.009708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:41.009819 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:41.009857 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 02:39:41.009949 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 02:39:41.010079 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":28,"thread_start_us":73,"threads_started":1}
I20260927 02:39:41.010229 11121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:41.004872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46038 (local address 127.0.98.2:44811)
0927 02:39:41.005303 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:41.005315 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:41.005338 (+ 23us) server_negotiation.cc:407] Connection header received
0927 02:39:41.005504 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:41.005510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:41.005732 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:41.005883 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:39:41.006347 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:41.007429 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:41.008812 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:41.009605 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:41.009690 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:41.009819 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:41.009986 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 02:39:41.010080 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":92,"threads_started":1}
I20260927 02:39:41.010226 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:41.004792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44245 (local address 127.0.0.1:40650)
0927 02:39:41.005642 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:41.005655 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:41.005742 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:41.006658 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:41.006661 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:41.006676 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:39:41.006765 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:41.006772 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:41.007708 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:41.007711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:41.008798 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:41.008806 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:41.009794 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:41.009797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:41.009886 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:41.009908 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 02:39:41.009971 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 02:39:41.010079 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":789,"mutex_wait_us":55,"thread_start_us":75,"threads_started":1}
I20260927 02:39:41.010582 11123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:41.005477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40650 (local address 127.0.98.3:44245)
0927 02:39:41.006294 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:41.006307 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:41.006328 (+ 21us) server_negotiation.cc:407] Connection header received
0927 02:39:41.006398 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:41.006403 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:41.006543 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:41.006668 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:39:41.006846 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:41.007576 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:41.008918 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:41.009488 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:41.009583 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:41.009735 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:41.010307 (+ 572us) server_negotiation.cc:299] Negotiation successful
0927 02:39:41.010403 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":692,"thread_start_us":89,"threads_started":1}
I20260927 02:39:41.011121 11124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:41.005847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44122 (local address 127.0.98.1:44833)
0927 02:39:41.006337 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:41.006349 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:41.006369 (+ 20us) server_negotiation.cc:407] Connection header received
0927 02:39:41.006436 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:41.006441 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:41.006623 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:41.006997 (+ 374us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:39:41.007129 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:41.008125 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:41.009554 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:41.010307 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:41.010835 (+ 528us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:41.010931 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:41.010967 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 02:39:41.011024 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":131,"threads_started":1}
I20260927 02:39:41.010752 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:41.005438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44833 (local address 127.0.0.1:44122)
0927 02:39:41.005995 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:41.006006 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:41.006077 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:41.006767 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:41.006770 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:41.006787 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:39:41.006871 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:41.006876 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:41.008263 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:41.008267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:41.009429 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:41.009440 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:41.010429 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:41.010433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:41.010514 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:41.010536 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 02:39:41.010585 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 02:39:41.010644 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":45,"thread_start_us":72,"threads_started":1}
I20260927 02:39:41.013688 10774 tablet_service.cc:1467] Processing CreateTablet for tablet 8053af9d71cd4446b113c74940e19f80 (DEFAULT_TABLE table=test-table [id=069bd9f1d9d543d183195074b5fa4099]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:39:41.013759 11042 tablet_service.cc:1467] Processing CreateTablet for tablet 8053af9d71cd4446b113c74940e19f80 (DEFAULT_TABLE table=test-table [id=069bd9f1d9d543d183195074b5fa4099]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:39:41.013690 10908 tablet_service.cc:1467] Processing CreateTablet for tablet 8053af9d71cd4446b113c74940e19f80 (DEFAULT_TABLE table=test-table [id=069bd9f1d9d543d183195074b5fa4099]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:39:41.014621 10774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8053af9d71cd4446b113c74940e19f80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:39:41.014676 11042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8053af9d71cd4446b113c74940e19f80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:39:41.014669 10908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8053af9d71cd4446b113c74940e19f80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:39:41.021183 11126 tablet_bootstrap.cc:492] T 8053af9d71cd4446b113c74940e19f80 P f82c119fe4284a839dd48d9795d642ec: Bootstrap starting.
I20260927 02:39:41.021658 11125 tablet_bootstrap.cc:492] T 8053af9d71cd4446b113c74940e19f80 P 78f99620825f4c73b79001e1d5c48053: Bootstrap starting.
I20260927 02:39:41.022514 11127 tablet_bootstrap.cc:492] T 8053af9d71cd4446b113c74940e19f80 P e93475820afa458d9b84e82aa55cd089: Bootstrap starting.
I20260927 02:39:41.023869 11126 tablet_bootstrap.cc:654] T 8053af9d71cd4446b113c74940e19f80 P f82c119fe4284a839dd48d9795d642ec: Neither blocks nor log segments found. Creating new log.
I20260927 02:39:41.023941 11125 tablet_bootstrap.cc:654] T 8053af9d71cd4446b113c74940e19f80 P 78f99620825f4c73b79001e1d5c48053: Neither blocks nor log segments found. Creating new log.
I20260927 02:39:41.024176 11127 tablet_bootstrap.cc:654] T 8053af9d71cd4446b113c74940e19f80 P e93475820afa458d9b84e82aa55cd089: Neither blocks nor log segments found. Creating new log.
I20260927 02:39:41.024726 11126 log.cc:824] T 8053af9d71cd4446b113c74940e19f80 P f82c119fe4284a839dd48d9795d642ec: Log is configured to *not* fsync() on all Append() calls
I20260927 02:39:41.024721 11125 log.cc:824] T 8053af9d71cd4446b113c74940e19f80 P 78f99620825f4c73b79001e1d5c48053: Log is configured to *not* fsync() on all Append() calls
I20260927 02:39:41.025121 11127 log.cc:824] T 8053af9d71cd4446b113c74940e19f80 P e93475820afa458d9b84e82aa55cd089: Log is configured to *not* fsync() on all Append() calls
I20260927 02:39:41.026634 11125 tablet_bootstrap.cc:492] T 8053af9d71cd4446b113c74940e19f80 P 78f99620825f4c73b79001e1d5c48053: No bootstrap required, opened a new log
I20260927 02:39:41.026875 11125 ts_tablet_manager.cc:1397] T 8053af9d71cd4446b113c74940e19f80 P 78f99620825f4c73b79001e1d5c48053: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 02:39:41.026938 11126 tablet_bootstrap.cc:492] T 8053af9d71cd4446b113c74940e19f80 P f82c119fe4284a839dd48d9795d642ec: No bootstrap required, opened a new log
I20260927 02:39:41.026996 11127 tablet_bootstrap.cc:492] T 8053af9d71cd4446b113c74940e19f80 P e93475820afa458d9b84e82aa55cd089: No bootstrap required, opened a new log
I20260927 02:39:41.027143 11126 ts_tablet_manager.cc:1397] T 8053af9d71cd4446b113c74940e19f80 P f82c119fe4284a839dd48d9795d642ec: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 02:39:41.027218 11127 ts_tablet_manager.cc:1397] T 8053af9d71cd4446b113c74940e19f80 P e93475820afa458d9b84e82aa55cd089: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260927 02:39:41.029999 11127 raft_consensus.cc:348] T 8053af9d71cd4446b113c74940e19f80 P e93475820afa458d9b84e82aa55cd089 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f82c119fe4284a839dd48d9795d642ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44811 } } peers { permanent_uuid: "e93475820afa458d9b84e82aa55cd089" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44245 } } peers { permanent_uuid: "78f99620825f4c73b79001e1d5c48053" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44833 } }
I20260927 02:39:41.030241 11127 raft_consensus.cc:374] T 8053af9d71cd4446b113c74940e19f80 P e93475820afa458d9b84e82aa55cd089 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:39:41.030319 11127 raft_consensus.cc:729] T 8053af9d71cd4446b113c74940e19f80 P e93475820afa458d9b84e82aa55cd089 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e93475820afa458d9b84e82aa55cd089, State: Initialized, Role: FOLLOWER
I20260927 02:39:41.030712 11127 consensus_queue.cc:260] T 8053af9d71cd4446b113c74940e19f80 P e93475820afa458d9b84e82aa55cd089 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f82c119fe4284a839dd48d9795d642ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44811 } } peers { permanent_uuid: "e93475820afa458d9b84e82aa55cd089" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44245 } } peers { permanent_uuid: "78f99620825f4c73b79001e1d5c48053" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44833 } }
I20260927 02:39:41.030642 11126 raft_consensus.cc:348] T 8053af9d71cd4446b113c74940e19f80 P f82c119fe4284a839dd48d9795d642ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f82c119fe4284a839dd48d9795d642ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44811 } } peers { permanent_uuid: "e93475820afa458d9b84e82aa55cd089" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44245 } } peers { permanent_uuid: "78f99620825f4c73b79001e1d5c48053" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44833 } }
I20260927 02:39:41.030893 11126 raft_consensus.cc:374] T 8053af9d71cd4446b113c74940e19f80 P f82c119fe4284a839dd48d9795d642ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:39:41.030786 11125 raft_consensus.cc:348] T 8053af9d71cd4446b113c74940e19f80 P 78f99620825f4c73b79001e1d5c48053 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f82c119fe4284a839dd48d9795d642ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44811 } } peers { permanent_uuid: "e93475820afa458d9b84e82aa55cd089" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44245 } } peers { permanent_uuid: "78f99620825f4c73b79001e1d5c48053" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44833 } }
I20260927 02:39:41.030936 11126 raft_consensus.cc:729] T 8053af9d71cd4446b113c74940e19f80 P f82c119fe4284a839dd48d9795d642ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f82c119fe4284a839dd48d9795d642ec, State: Initialized, Role: FOLLOWER
I20260927 02:39:41.030988 11125 raft_consensus.cc:374] T 8053af9d71cd4446b113c74940e19f80 P 78f99620825f4c73b79001e1d5c48053 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:39:41.031045 11125 raft_consensus.cc:729] T 8053af9d71cd4446b113c74940e19f80 P 78f99620825f4c73b79001e1d5c48053 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78f99620825f4c73b79001e1d5c48053, State: Initialized, Role: FOLLOWER
I20260927 02:39:41.031390 11108 heartbeater.cc:503] Master 127.0.98.62:39563 was elected leader, sending a full tablet report...
I20260927 02:39:41.031379 11126 consensus_queue.cc:260] T 8053af9d71cd4446b113c74940e19f80 P f82c119fe4284a839dd48d9795d642ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f82c119fe4284a839dd48d9795d642ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44811 } } peers { permanent_uuid: "e93475820afa458d9b84e82aa55cd089" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44245 } } peers { permanent_uuid: "78f99620825f4c73b79001e1d5c48053" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44833 } }
I20260927 02:39:41.031387 11125 consensus_queue.cc:260] T 8053af9d71cd4446b113c74940e19f80 P 78f99620825f4c73b79001e1d5c48053 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f82c119fe4284a839dd48d9795d642ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44811 } } peers { permanent_uuid: "e93475820afa458d9b84e82aa55cd089" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44245 } } peers { permanent_uuid: "78f99620825f4c73b79001e1d5c48053" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44833 } }
I20260927 02:39:41.032178 10840 heartbeater.cc:503] Master 127.0.98.62:39563 was elected leader, sending a full tablet report...
I20260927 02:39:41.032311 11125 ts_tablet_manager.cc:1428] T 8053af9d71cd4446b113c74940e19f80 P 78f99620825f4c73b79001e1d5c48053: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260927 02:39:41.032310 11126 ts_tablet_manager.cc:1428] T 8053af9d71cd4446b113c74940e19f80 P f82c119fe4284a839dd48d9795d642ec: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 02:39:41.032349 10974 heartbeater.cc:503] Master 127.0.98.62:39563 was elected leader, sending a full tablet report...
I20260927 02:39:41.033506 11127 ts_tablet_manager.cc:1428] T 8053af9d71cd4446b113c74940e19f80 P e93475820afa458d9b84e82aa55cd089: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
W20260927 02:39:41.052639 10841 tablet.cc:2369] T 8053af9d71cd4446b113c74940e19f80 P 78f99620825f4c73b79001e1d5c48053: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:39:41.103622 11133 raft_consensus.cc:482] T 8053af9d71cd4446b113c74940e19f80 P f82c119fe4284a839dd48d9795d642ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:39:41.103807 11133 raft_consensus.cc:504] T 8053af9d71cd4446b113c74940e19f80 P f82c119fe4284a839dd48d9795d642ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f82c119fe4284a839dd48d9795d642ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44811 } } peers { permanent_uuid: "e93475820afa458d9b84e82aa55cd089" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44245 } } peers { permanent_uuid: "78f99620825f4c73b79001e1d5c48053" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44833 } }
I20260927 02:39:41.104904 11133 leader_election.cc:288] T 8053af9d71cd4446b113c74940e19f80 P f82c119fe4284a839dd48d9795d642ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e93475820afa458d9b84e82aa55cd089 (127.0.98.3:44245), 78f99620825f4c73b79001e1d5c48053 (127.0.98.1:44833)
I20260927 02:39:41.109803 11124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:41.105280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58741 (local address 127.0.98.1:44833)
0927 02:39:41.105407 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:41.105412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:41.105609 (+ 197us) server_negotiation.cc:407] Connection header received
0927 02:39:41.105735 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:41.105738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:41.105780 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:41.105851 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:39:41.106166 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:41.106938 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:41.108559 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:41.109167 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:41.109272 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:41.109336 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:41.109593 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 02:39:41.109661 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 02:39:41.109802 11123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:41.105199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47865 (local address 127.0.98.3:44245)
0927 02:39:41.105325 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:41.105330 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:41.105487 (+ 157us) server_negotiation.cc:407] Connection header received
0927 02:39:41.105673 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:41.105676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:41.105731 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:41.105791 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:39:41.106081 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:41.106938 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:41.108559 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:41.109167 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:41.109279 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:41.109336 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:41.109593 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 02:39:41.109662 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 02:39:41.109845 10976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:41.105116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44245 (local address 127.0.98.2:47865)
0927 02:39:41.105412 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:41.105427 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:41.105511 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:41.105840 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:41.105844 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:41.105859 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:39:41.105974 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:41.105981 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:41.107051 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:41.107054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:41.108442 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:41.108452 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:41.109290 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:41.109294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:41.109483 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:41.109500 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 02:39:41.109570 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 02:39:41.109676 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":100}
I20260927 02:39:41.109855 11134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:41.105199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44833 (local address 127.0.98.2:58741)
0927 02:39:41.105510 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:41.105524 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:39:41.105638 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:39:41.105928 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:39:41.105931 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:39:41.105949 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:39:41.106056 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:41.106061 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:41.107051 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:41.107055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:41.108442 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:39:41.108452 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:39:41.109290 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:39:41.109294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:39:41.109463 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:41.109492 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 02:39:41.109566 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 02:39:41.109665 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":156,"thread_start_us":109,"threads_started":1}
I20260927 02:39:41.110684 10794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8053af9d71cd4446b113c74940e19f80" candidate_uuid: "f82c119fe4284a839dd48d9795d642ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78f99620825f4c73b79001e1d5c48053" is_pre_election: true
I20260927 02:39:41.110688 11062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8053af9d71cd4446b113c74940e19f80" candidate_uuid: "f82c119fe4284a839dd48d9795d642ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e93475820afa458d9b84e82aa55cd089" is_pre_election: true
I20260927 02:39:41.110998 10794 raft_consensus.cc:2454] T 8053af9d71cd4446b113c74940e19f80 P 78f99620825f4c73b79001e1d5c48053 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f82c119fe4284a839dd48d9795d642ec in term 0.
I20260927 02:39:41.110998 11062 raft_consensus.cc:2454] T 8053af9d71cd4446b113c74940e19f80 P e93475820afa458d9b84e82aa55cd089 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f82c119fe4284a839dd48d9795d642ec in term 0.
I20260927 02:39:41.111464 10864 leader_election.cc:302] T 8053af9d71cd4446b113c74940e19f80 P f82c119fe4284a839dd48d9795d642ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78f99620825f4c73b79001e1d5c48053, f82c119fe4284a839dd48d9795d642ec; no voters:
I20260927 02:39:41.111761 11133 raft_consensus.cc:2781] T 8053af9d71cd4446b113c74940e19f80 P f82c119fe4284a839dd48d9795d642ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:39:41.111842 11133 raft_consensus.cc:482] T 8053af9d71cd4446b113c74940e19f80 P f82c119fe4284a839dd48d9795d642ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:39:41.111887 11133 raft_consensus.cc:3037] T 8053af9d71cd4446b113c74940e19f80 P f82c119fe4284a839dd48d9795d642ec [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:39:41.112914 11133 raft_consensus.cc:504] T 8053af9d71cd4446b113c74940e19f80 P f82c119fe4284a839dd48d9795d642ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f82c119fe4284a839dd48d9795d642ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44811 } } peers { permanent_uuid: "e93475820afa458d9b84e82aa55cd089" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44245 } } peers { permanent_uuid: "78f99620825f4c73b79001e1d5c48053" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44833 } }
I20260927 02:39:41.113243 11133 leader_election.cc:288] T 8053af9d71cd4446b113c74940e19f80 P f82c119fe4284a839dd48d9795d642ec [CANDIDATE]: Term 1 election: Requested vote from peers e93475820afa458d9b84e82aa55cd089 (127.0.98.3:44245), 78f99620825f4c73b79001e1d5c48053 (127.0.98.1:44833)
I20260927 02:39:41.113570 11062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8053af9d71cd4446b113c74940e19f80" candidate_uuid: "f82c119fe4284a839dd48d9795d642ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e93475820afa458d9b84e82aa55cd089"
I20260927 02:39:41.113612 10794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8053af9d71cd4446b113c74940e19f80" candidate_uuid: "f82c119fe4284a839dd48d9795d642ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78f99620825f4c73b79001e1d5c48053"
I20260927 02:39:41.113687 11062 raft_consensus.cc:3037] T 8053af9d71cd4446b113c74940e19f80 P e93475820afa458d9b84e82aa55cd089 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:39:41.113832 10794 raft_consensus.cc:3037] T 8053af9d71cd4446b113c74940e19f80 P 78f99620825f4c73b79001e1d5c48053 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:39:41.114728 11062 raft_consensus.cc:2454] T 8053af9d71cd4446b113c74940e19f80 P e93475820afa458d9b84e82aa55cd089 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f82c119fe4284a839dd48d9795d642ec in term 1.
I20260927 02:39:41.114890 10794 raft_consensus.cc:2454] T 8053af9d71cd4446b113c74940e19f80 P 78f99620825f4c73b79001e1d5c48053 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f82c119fe4284a839dd48d9795d642ec in term 1.
I20260927 02:39:41.115002 10863 leader_election.cc:302] T 8053af9d71cd4446b113c74940e19f80 P f82c119fe4284a839dd48d9795d642ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e93475820afa458d9b84e82aa55cd089, f82c119fe4284a839dd48d9795d642ec; no voters:
I20260927 02:39:41.115162 11133 raft_consensus.cc:2781] T 8053af9d71cd4446b113c74940e19f80 P f82c119fe4284a839dd48d9795d642ec [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:39:41.115365 11133 raft_consensus.cc:686] T 8053af9d71cd4446b113c74940e19f80 P f82c119fe4284a839dd48d9795d642ec [term 1 LEADER]: Becoming Leader. State: Replica: f82c119fe4284a839dd48d9795d642ec, State: Running, Role: LEADER
I20260927 02:39:41.115727 11133 consensus_queue.cc:237] T 8053af9d71cd4446b113c74940e19f80 P f82c119fe4284a839dd48d9795d642ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f82c119fe4284a839dd48d9795d642ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44811 } } peers { permanent_uuid: "e93475820afa458d9b84e82aa55cd089" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44245 } } peers { permanent_uuid: "78f99620825f4c73b79001e1d5c48053" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44833 } }
I20260927 02:39:41.118726 10649 catalog_manager.cc:5495] T 8053af9d71cd4446b113c74940e19f80 P f82c119fe4284a839dd48d9795d642ec reported cstate change: term changed from 0 to 1, leader changed from <none> to f82c119fe4284a839dd48d9795d642ec (127.0.98.2). New cstate: current_term: 1 leader_uuid: "f82c119fe4284a839dd48d9795d642ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f82c119fe4284a839dd48d9795d642ec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e93475820afa458d9b84e82aa55cd089" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78f99620825f4c73b79001e1d5c48053" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44833 } health_report { overall_health: UNKNOWN } } }
I20260927 02:39:41.131516 11121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:41.128288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46040 (local address 127.0.98.2:44811)
0927 02:39:41.128381 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 02:39:41.128385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:39:41.128396 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:39:41.128507 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:39:41.128510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:39:41.128559 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:39:41.128656 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:39:41.128830 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:41.129378 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:41.130283 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:39:41.130484 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:39:41.130555 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:39:41.130781 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:39:41.131159 (+ 378us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:39:41.131251 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 02:39:41.131358 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 02:39:41.131401 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 02:39:41.136673 10794 raft_consensus.cc:1261] T 8053af9d71cd4446b113c74940e19f80 P 78f99620825f4c73b79001e1d5c48053 [term 1 FOLLOWER]: Refusing update from remote peer f82c119fe4284a839dd48d9795d642ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:39:41.136674 11062 raft_consensus.cc:1261] T 8053af9d71cd4446b113c74940e19f80 P e93475820afa458d9b84e82aa55cd089 [term 1 FOLLOWER]: Refusing update from remote peer f82c119fe4284a839dd48d9795d642ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:39:41.137331 11135 consensus_queue.cc:1035] T 8053af9d71cd4446b113c74940e19f80 P f82c119fe4284a839dd48d9795d642ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "e93475820afa458d9b84e82aa55cd089" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:39:41.137486 11133 consensus_queue.cc:1035] T 8053af9d71cd4446b113c74940e19f80 P f82c119fe4284a839dd48d9795d642ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "78f99620825f4c73b79001e1d5c48053" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:39:41.153144 10649 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:39:41.153302 10649 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:39:41.155483 10649 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 274fca3ba5704aa49474b3fe1248af72: Sending DeleteTablet for 3 replicas of tablet 8053af9d71cd4446b113c74940e19f80
I20260927 02:39:41.156205 10908 tablet_service.cc:1514] Processing DeleteTablet for tablet 8053af9d71cd4446b113c74940e19f80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:39:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46038
I20260927 02:39:41.156205 10774 tablet_service.cc:1514] Processing DeleteTablet for tablet 8053af9d71cd4446b113c74940e19f80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:39:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44122
I20260927 02:39:41.156358 11042 tablet_service.cc:1514] Processing DeleteTablet for tablet 8053af9d71cd4446b113c74940e19f80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:39:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40650
I20260927 02:39:41.156644 11149 tablet_replica.cc:330] T 8053af9d71cd4446b113c74940e19f80 P 78f99620825f4c73b79001e1d5c48053: stopping tablet replica
I20260927 02:39:41.156827 11150 tablet_replica.cc:330] T 8053af9d71cd4446b113c74940e19f80 P e93475820afa458d9b84e82aa55cd089: stopping tablet replica
I20260927 02:39:41.156944 11149 raft_consensus.cc:2229] T 8053af9d71cd4446b113c74940e19f80 P 78f99620825f4c73b79001e1d5c48053 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:39:41.157054 11150 raft_consensus.cc:2229] T 8053af9d71cd4446b113c74940e19f80 P e93475820afa458d9b84e82aa55cd089 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:39:41.157224 11149 raft_consensus.cc:2258] T 8053af9d71cd4446b113c74940e19f80 P 78f99620825f4c73b79001e1d5c48053 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:39:41.157274 11150 raft_consensus.cc:2258] T 8053af9d71cd4446b113c74940e19f80 P e93475820afa458d9b84e82aa55cd089 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:39:41.157612 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 10711
I20260927 02:39:41.157644 11148 tablet_replica.cc:330] T 8053af9d71cd4446b113c74940e19f80 P f82c119fe4284a839dd48d9795d642ec: stopping tablet replica
I20260927 02:39:41.157879 11148 raft_consensus.cc:2229] T 8053af9d71cd4446b113c74940e19f80 P f82c119fe4284a839dd48d9795d642ec [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:39:41.158516 11148 raft_consensus.cc:2258] T 8053af9d71cd4446b113c74940e19f80 P f82c119fe4284a839dd48d9795d642ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:39:41.159678 11150 ts_tablet_manager.cc:1905] T 8053af9d71cd4446b113c74940e19f80 P e93475820afa458d9b84e82aa55cd089: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:39:41.162199 11150 ts_tablet_manager.cc:1918] T 8053af9d71cd4446b113c74940e19f80 P e93475820afa458d9b84e82aa55cd089: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:39:41.162297 11150 log.cc:1196] T 8053af9d71cd4446b113c74940e19f80 P e93475820afa458d9b84e82aa55cd089: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/ts-2/wal/wals/8053af9d71cd4446b113c74940e19f80
I20260927 02:39:41.162598 11150 ts_tablet_manager.cc:1939] T 8053af9d71cd4446b113c74940e19f80 P e93475820afa458d9b84e82aa55cd089: Deleting consensus metadata
I20260927 02:39:41.163201 11148 ts_tablet_manager.cc:1905] T 8053af9d71cd4446b113c74940e19f80 P f82c119fe4284a839dd48d9795d642ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:39:41.164072 10635 catalog_manager.cc:4841] TS e93475820afa458d9b84e82aa55cd089 (127.0.98.3:44245): tablet 8053af9d71cd4446b113c74940e19f80 (table test-table [id=069bd9f1d9d543d183195074b5fa4099]) successfully deleted
W20260927 02:39:41.164827 10636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44833 (error 108)
I20260927 02:39:41.165604 11148 ts_tablet_manager.cc:1918] T 8053af9d71cd4446b113c74940e19f80 P f82c119fe4284a839dd48d9795d642ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:39:41.165680 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:41.165438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44833 (local address 127.0.0.1:44134)
0927 02:39:41.165541 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:41.165598 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 02:39:41.165714 11148 log.cc:1196] T 8053af9d71cd4446b113c74940e19f80 P f82c119fe4284a839dd48d9795d642ec: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790476718023131-392-0/minicluster-data/ts-1/wal/wals/8053af9d71cd4446b113c74940e19f80
I20260927 02:39:41.165766 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 10845
I20260927 02:39:41.165990 11148 ts_tablet_manager.cc:1939] T 8053af9d71cd4446b113c74940e19f80 P f82c119fe4284a839dd48d9795d642ec: Deleting consensus metadata
W20260927 02:39:41.167073 10636 catalog_manager.cc:4584] TS 78f99620825f4c73b79001e1d5c48053 (127.0.98.1:44833): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8053af9d71cd4446b113c74940e19f80: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44833: connect: Connection refused (error 111)
I20260927 02:39:41.170629 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 10979
I20260927 02:39:41.172379 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:39:41.170724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44811 (local address 127.0.0.1:46052)
0927 02:39:41.172252 (+ 1528us) negotiation.cc:107] Waiting for socket to connect
0927 02:39:41.172307 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1447}
I20260927 02:39:41.175422 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 10617
2026-09-27T02:39:41Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2175 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:39:41 dist-test-slave-zv8z krb5kdc[11154](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:39:41 dist-test-slave-zv8z krb5kdc[11154](info): set up 2 sockets
Sep 27 02:39:41 dist-test-slave-zv8z krb5kdc[11154](info): commencing operation
krb5kdc: starting...
W20260927 02:39:42.126904 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/krb5kdc/test-user.keytab.
Sep 27 02:39:42 dist-test-slave-zv8z krb5kdc[11154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:39:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:39:42Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260927 02:39:42.282775 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskBFiRWZ/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/postgres -l logfile start
2026-09-27 02:39:45.251 UTC [11184] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 02:39:45.251 UTC [11184] LOG: listening on IPv4 address "127.0.98.20", port 51909
2026-09-27 02:39:45.254 UTC [11184] LOG: listening on Unix socket "/tmp/.s.PGSQL.51909"
2026-09-27 02:39:45.280 UTC [11187] LOG: database system was shut down at 2026-09-27 02:39:43 UTC
2026-09-27 02:39:45.284 UTC [11184] LOG: database system is ready to accept connections
I20260927 02:39:46.162715 392 mini_postgres.cc:96] Postgres bound to 51909
2026-09-27 02:39:46.168 UTC [11195] FATAL: database "slave" does not exist
127.0.98.20:51909 - accepting connections
I20260927 02:39:46.169185 392 mini_ranger.cc:161] Starting Ranger...
I20260927 02:39:46.183262 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 02:39:46.774166 392 mini_ranger.cc:87] Created ranger Postgres database
I20260927 02:39:46.774263 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ranger-admin
2026-09-27 02:39:47,044 [I] DB FLAVOR :POSTGRES
2026-09-27 02:39:47,044 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 02:39:47,044 [I] Checking connection..
2026-09-27 02:39:47,044 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBFiRWZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBFiRWZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 02:39:47,273 [I] Checking connection passed.
2026-09-27 02:39:47,273 [I] --------- Verifying version history table ---------
2026-09-27 02:39:47,273 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBFiRWZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBFiRWZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 02:39:47,520 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 02:39:47,521 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBFiRWZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBFiRWZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 02:39:47,749 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 02:39:47,750 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 02:39:47,750 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBFiRWZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBFiRWZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51909/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBFiRWZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 02:39:47.987 UTC [11259] WARNING: there is no transaction in progress
2026-09-27 02:39:47,993 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 02:39:47,993 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBFiRWZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBFiRWZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 02:39:48,241 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 02:39:48,242 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 02:39:48,242 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBFiRWZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBFiRWZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 02:39:48,467 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBFiRWZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBFiRWZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 02:39:48,715 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBFiRWZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBFiRWZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-zv8z.c.gcp-upstream.internal','N') ;"
2026-09-27 02:39:48,930 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBFiRWZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBFiRWZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 02:39:49,176 [I] Table x_portal_user does not exist in database ranger
2026-09-27 02:39:49,177 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBFiRWZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBFiRWZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 02:39:49,432 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 02:39:49,432 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBFiRWZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBFiRWZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 02:39:49,680 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBFiRWZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBFiRWZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 02:39:49,900 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 02:39:49,900 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBFiRWZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBFiRWZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51909/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBFiRWZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 02:39:50.842 UTC [11367] WARNING: there is no transaction in progress
2026-09-27 02:39:50.858 UTC [11367] WARNING: there is no transaction in progress
2026-09-27 02:39:50.874 UTC [11367] WARNING: there is no transaction in progress
2026-09-27 02:39:50.890 UTC [11367] WARNING: there is no transaction in progress
2026-09-27 02:39:50.905 UTC [11367] WARNING: there is no transaction in progress
2026-09-27 02:39:50.922 UTC [11367] WARNING: there is no transaction in progress
2026-09-27 02:39:50.974 UTC [11367] WARNING: there is no transaction in progress
2026-09-27 02:39:50.981 UTC [11367] WARNING: there is no transaction in progress
2026-09-27 02:39:50.995 UTC [11367] WARNING: there is no transaction in progress
2026-09-27 02:39:51.005 UTC [11367] WARNING: there is no transaction in progress
2026-09-27 02:39:51.014 UTC [11367] WARNING: there is no transaction in progress
2026-09-27 02:39:51.028 UTC [11367] WARNING: there is no transaction in progress
2026-09-27 02:39:51.037 UTC [11367] WARNING: there is no transaction in progress
2026-09-27 02:39:51.044 UTC [11367] WARNING: there is no transaction in progress
2026-09-27 02:39:51.050 UTC [11367] WARNING: there is no transaction in progress
2026-09-27 02:39:51.057 UTC [11367] WARNING: there is no transaction in progress
2026-09-27 02:39:51,669 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 02:39:51,670 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBFiRWZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBFiRWZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 02:39:51,900 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 02:39:51,901 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBFiRWZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBFiRWZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 02:39:52,136 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 02:39:52,136 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBFiRWZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBFiRWZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 02:39:52,369 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 02:39:52,370 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBFiRWZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBFiRWZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 02:39:52,618 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBFiRWZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBFiRWZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 02:39:52,857 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBFiRWZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBFiRWZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-zv8z.c.gcp-upstream.internal';"
2026-09-27 02:39:53,072 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 02:39:53,073 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBFiRWZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBFiRWZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 02:39:53,301 [I] DB_PATCHES have already been applied
I20260927 02:39:53.306872 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskBFiRWZ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskBFiRWZ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskBFiRWZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBFiRWZ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 02:39:53.306938 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260927 02:39:53.310039 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:58229
Sep 27, 2026 2:39:53 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 2:39:53 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 2:39:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 2:39:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 2:39:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 2:39:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 2:39:54 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 02:39:54.309393 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.140s user 0.000s sys 0.009s
Sep 27, 2026 2:40:14 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 2:40:14 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 2:40:14 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 2:40:14 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 2:40:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 2:40:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 2:40:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 2:40:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 2:40:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 2:40:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 2:40:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 2:40:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 2:40:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 2:40:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 2:40:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 2:40:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 2:40:14 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 2:40:15 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 02:40:15.838781 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 02:40:15 dist-test-slave-zv8z krb5kdc[11154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476815, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 02:40:15.934226 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 02:40:15.942893 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 02:40:16.788697 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 02:40:16.788793 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ranger-kms
I20260927 02:40:16.788875 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:51909
2026-09-27 02:40:16,957 [I] DB FLAVOR :POSTGRES
2026-09-27 02:40:16,958 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 02:40:16,958 [I] Checking connection
2026-09-27 02:40:16,958 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBFiRWZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51909/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 02:40:17,182 [I] connection success
2026-09-27 02:40:17,182 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 02:40:17,183 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 02:40:17,183 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBFiRWZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51909/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 02:40:17,409 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 02:40:17,409 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 02:40:17,409 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 02:40:17,409 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBFiRWZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51909/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 02:40:17,656 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 02:40:18.100966 392 mini_ranger_kms.cc:310] Created kms service
I20260927 02:40:18.283478 392 mini_ranger_kms.cc:326] Created kudu user
I20260927 02:40:18.323398 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 02:40:18 dist-test-slave-zv8z krb5kdc[11154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476815, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:40:18.692561 392 mini_ranger_kms.cc:384] Added ranger policy
I20260927 02:40:18.692715 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskBFiRWZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBFiRWZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskBFiRWZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 02:40:18.692752 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260927 02:40:18.696308 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11547
I20260927 02:40:18.696431 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:39187
Sep 27, 2026 2:40:19 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 2:40:19 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 2:40:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 2:40:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 2:40:20 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 2:40:20 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 02:40:20.782074 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.848s user 0.013s sys 0.141s
I20260927 02:40:20.782195 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 02:40:20.782415 392 mini_ranger_kms.cc:401] 127.0.98.20:39187/kms/v1/keys
Sep 27 02:40:20 dist-test-slave-zv8z krb5kdc[11154](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790476820, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 02:40:21 dist-test-slave-zv8z krb5kdc[11154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476820, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 02:40:21 dist-test-slave-zv8z krb5kdc[11154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476820, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 02:40:23 dist-test-slave-zv8z krb5kdc[11154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:40:23.813040 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41825
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41825
--ranger_config_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39187/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:40:23.923974 11598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:40:23.924238 11598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:40:23.924275 11598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:40:23.928324 11598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:40:23.928378 11598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:40:23.928397 11598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:40:23.928411 11598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:40:23.928426 11598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:40:23.933799 11598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:39187/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41825
--ranger_config_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:40:23.934885 11598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:40:23.935909 11598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:40:23.944507 11604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:40:23.944645 11598 server_base.cc:1034] running on GCE node
W20260927 02:40:23.944772 11606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:40:23.944517 11603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:40:23.945417 11598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:40:23.946521 11598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:40:23.947707 11598 hybrid_clock.cc:648] HybridClock initialized: now 1790476823947686 us; error 38 us; skew 500 ppm
Sep 27 02:40:23 dist-test-slave-zv8z krb5kdc[11154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:40:23.950538 11598 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:40:23.951537 11598 webserver.cc:460] Webserver started at http://127.0.98.62:36631/ using document root <none> and password file <none>
I20260927 02:40:23.952157 11598 fs_manager.cc:359] Metadata directory not provided
I20260927 02:40:23.952232 11598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:40:23.952440 11598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 02:40:23 dist-test-slave-zv8z krb5kdc[11154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:40:24.004598 11598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/master-0/data/instance:
uuid: "e0060233a8534cb4909fd0dcab01de84"
format_stamp: "Formatted at 2026-09-27 02:40:24 on dist-test-slave-zv8z"
server_key: "9e36212ddd9f884ece13ee9f8096f727"
server_key_iv: "37055990f8ff9e327382ca6fab7cf58a"
server_key_version: "kuduclusterkey@0"
I20260927 02:40:24.005185 11598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/master-0/wal/instance:
uuid: "e0060233a8534cb4909fd0dcab01de84"
format_stamp: "Formatted at 2026-09-27 02:40:24 on dist-test-slave-zv8z"
server_key: "9e36212ddd9f884ece13ee9f8096f727"
server_key_iv: "37055990f8ff9e327382ca6fab7cf58a"
server_key_version: "kuduclusterkey@0"
I20260927 02:40:24.009230 11598 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 02:40:24.026371 11613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:40:24.027779 11598 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:40:24.027892 11598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "e0060233a8534cb4909fd0dcab01de84"
format_stamp: "Formatted at 2026-09-27 02:40:24 on dist-test-slave-zv8z"
server_key: "9e36212ddd9f884ece13ee9f8096f727"
server_key_iv: "37055990f8ff9e327382ca6fab7cf58a"
server_key_version: "kuduclusterkey@0"
I20260927 02:40:24.028010 11598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:40:24.050134 11598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:40:24.052951 11598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:40:24.053123 11598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:40:24.060936 11598 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41825
I20260927 02:40:24.060979 11665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41825 every 8 connection(s)
I20260927 02:40:24.061908 11598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:40:24.064617 11666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:40:24.069689 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 11598
I20260927 02:40:24.069808 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/master-0/wal/instance
I20260927 02:40:24.070096 392 external_mini_cluster.cc:1428] Setting key b41c0b07f7b5a264e439c4b5aabcdd0d
I20260927 02:40:24.070313 11666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0060233a8534cb4909fd0dcab01de84: Bootstrap starting.
I20260927 02:40:24.072824 11666 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0060233a8534cb4909fd0dcab01de84: Neither blocks nor log segments found. Creating new log.
I20260927 02:40:24.073539 11666 log.cc:824] T 00000000000000000000000000000000 P e0060233a8534cb4909fd0dcab01de84: Log is configured to *not* fsync() on all Append() calls
I20260927 02:40:24.075941 11666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0060233a8534cb4909fd0dcab01de84: No bootstrap required, opened a new log
Sep 27 02:40:24 dist-test-slave-zv8z krb5kdc[11154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:40:24.080272 11666 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0060233a8534cb4909fd0dcab01de84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0060233a8534cb4909fd0dcab01de84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41825 } }
I20260927 02:40:24.080540 11666 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0060233a8534cb4909fd0dcab01de84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:40:24.080631 11666 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0060233a8534cb4909fd0dcab01de84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0060233a8534cb4909fd0dcab01de84, State: Initialized, Role: FOLLOWER
I20260927 02:40:24.081125 11666 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0060233a8534cb4909fd0dcab01de84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0060233a8534cb4909fd0dcab01de84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41825 } }
I20260927 02:40:24.081275 11666 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0060233a8534cb4909fd0dcab01de84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:40:24.081357 11666 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0060233a8534cb4909fd0dcab01de84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:40:24.081450 11666 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0060233a8534cb4909fd0dcab01de84 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:40:24.082541 11666 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0060233a8534cb4909fd0dcab01de84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0060233a8534cb4909fd0dcab01de84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41825 } }
I20260927 02:40:24.082897 11666 leader_election.cc:302] T 00000000000000000000000000000000 P e0060233a8534cb4909fd0dcab01de84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0060233a8534cb4909fd0dcab01de84; no voters:
I20260927 02:40:24.083253 11666 leader_election.cc:288] T 00000000000000000000000000000000 P e0060233a8534cb4909fd0dcab01de84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:40:24.083345 11671 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0060233a8534cb4909fd0dcab01de84 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:40:24.083595 11671 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0060233a8534cb4909fd0dcab01de84 [term 1 LEADER]: Becoming Leader. State: Replica: e0060233a8534cb4909fd0dcab01de84, State: Running, Role: LEADER
I20260927 02:40:24.083935 11671 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0060233a8534cb4909fd0dcab01de84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0060233a8534cb4909fd0dcab01de84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41825 } }
I20260927 02:40:24.084362 11669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:40:24.071539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34242 (local address 127.0.98.62:41825)
0927 02:40:24.072016 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0927 02:40:24.072034 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 02:40:24.072076 (+ 42us) server_negotiation.cc:407] Connection header received
0927 02:40:24.072899 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:40:24.072921 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:40:24.073338 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:40:24.073691 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:40:24.074589 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:40:24.075591 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:40:24.076478 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:40:24.076948 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:40:24.077085 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:40:24.079439 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:40:24.079444 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:40:24.079458 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:40:24.079483 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:40:24.081080 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:40:24.081518 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:40:24.081524 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:40:24.081538 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:40:24.081641 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:40:24.081892 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:40:24.081895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:40:24.081897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:40:24.082319 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:40:24.082503 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 02:40:24.083128 (+ 625us) server_negotiation.cc:299] Negotiation successful
0927 02:40:24.083389 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":147,"threads_started":1}
I20260927 02:40:24.084517 11666 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0060233a8534cb4909fd0dcab01de84 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:40:24.085134 11671 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0060233a8534cb4909fd0dcab01de84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0060233a8534cb4909fd0dcab01de84. Latest consensus state: current_term: 1 leader_uuid: "e0060233a8534cb4909fd0dcab01de84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0060233a8534cb4909fd0dcab01de84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41825 } } }
I20260927 02:40:24.085242 11671 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0060233a8534cb4909fd0dcab01de84 [sys.catalog]: This master's current role is: LEADER
I20260927 02:40:24.085561 11675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:40:24.087550 11672 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0060233a8534cb4909fd0dcab01de84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0060233a8534cb4909fd0dcab01de84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0060233a8534cb4909fd0dcab01de84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41825 } } }
I20260927 02:40:24.087697 11672 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0060233a8534cb4909fd0dcab01de84 [sys.catalog]: This master's current role is: LEADER
I20260927 02:40:24.091480 11675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:40:24.091907 11666 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 02:40:24.103385 11675 catalog_manager.cc:1347] Generated new cluster ID: b4b721d0d0344d608a468763c47dc97c
I20260927 02:40:24.103473 11675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:40:24.123011 11675 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:40:24.124814 11675 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:40:24.139043 11675 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0060233a8534cb4909fd0dcab01de84: Generated new TSK 0
I20260927 02:40:24.140002 11675 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:40:24.873795 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39187/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41825
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:40:25.033720 11711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:40:25.034091 11711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:40:25.034175 11711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:40:25.040251 11711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:40:25.040337 11711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:40:25.040426 11711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:40:25.048435 11711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:39187/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:40:25.049868 11711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:40:25.051334 11711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:40:25.063795 11719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:40:25.064477 11718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:40:25.065126 11711 server_base.cc:1034] running on GCE node
W20260927 02:40:25.065372 11721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:40:25.065831 11711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:40:25.066586 11711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:40:25.067749 11711 hybrid_clock.cc:648] HybridClock initialized: now 1790476825067717 us; error 46 us; skew 500 ppm
Sep 27 02:40:25 dist-test-slave-zv8z krb5kdc[11154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:40:25.071455 11711 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:40:25.072605 11711 webserver.cc:460] Webserver started at http://127.0.98.1:44991/ using document root <none> and password file <none>
I20260927 02:40:25.073320 11711 fs_manager.cc:359] Metadata directory not provided
I20260927 02:40:25.073424 11711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:40:25.073660 11711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 02:40:25 dist-test-slave-zv8z krb5kdc[11154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:40:25.090272 11711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ts-0/data/instance:
uuid: "b5dffd2d01b04826bbcb2d0f43415b2a"
format_stamp: "Formatted at 2026-09-27 02:40:25 on dist-test-slave-zv8z"
server_key: "d2cabd9213d6da126cc4f8a7a2cd01e4"
server_key_iv: "71f7026a9a7ecef5cf6dc8ea1806bc59"
server_key_version: "kuduclusterkey@0"
I20260927 02:40:25.092986 11711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b5dffd2d01b04826bbcb2d0f43415b2a"
format_stamp: "Formatted at 2026-09-27 02:40:25 on dist-test-slave-zv8z"
server_key: "d2cabd9213d6da126cc4f8a7a2cd01e4"
server_key_iv: "71f7026a9a7ecef5cf6dc8ea1806bc59"
server_key_version: "kuduclusterkey@0"
I20260927 02:40:25.104941 11711 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.001s sys 0.003s
I20260927 02:40:25.131810 11728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:40:25.133665 11711 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.000s sys 0.003s
I20260927 02:40:25.133821 11711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ts-0/wal
uuid: "b5dffd2d01b04826bbcb2d0f43415b2a"
format_stamp: "Formatted at 2026-09-27 02:40:25 on dist-test-slave-zv8z"
server_key: "d2cabd9213d6da126cc4f8a7a2cd01e4"
server_key_iv: "71f7026a9a7ecef5cf6dc8ea1806bc59"
server_key_version: "kuduclusterkey@0"
I20260927 02:40:25.133980 11711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:40:25.158993 11711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:40:25.168267 11711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:40:25.168540 11711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:40:25.169327 11711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:40:25.170454 11711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:40:25.170560 11711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:40:25.170707 11711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:40:25.170784 11711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:40:25.181380 11711 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37391
I20260927 02:40:25.182677 11711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:40:25.186894 11843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37391 every 8 connection(s)
I20260927 02:40:25.192505 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 11711
I20260927 02:40:25.192647 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:40:25.192961 392 external_mini_cluster.cc:1428] Setting key f8e097b839fcf03846eed28d88e72bce
Sep 27 02:40:25 dist-test-slave-zv8z krb5kdc[11154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:40:25.206144 11847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:40:25.190396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41825 (local address 127.0.98.1:60045)
0927 02:40:25.191561 (+ 1165us) negotiation.cc:107] Waiting for socket to connect
0927 02:40:25.191619 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 02:40:25.192638 (+ 1019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:40:25.193543 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:40:25.193552 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:40:25.193877 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:40:25.194586 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:40:25.194607 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:40:25.195722 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:40:25.195725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:40:25.197028 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:40:25.197038 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:40:25.197773 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:40:25.197777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:40:25.197971 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:40:25.198684 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:40:25.198704 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:40:25.200421 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:40:25.202713 (+ 2292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:40:25.202720 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:40:25.202736 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:40:25.203109 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:40:25.203422 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:40:25.203425 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:40:25.203427 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:40:25.203627 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:40:25.204136 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:40:25.204143 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:40:25.204485 (+ 342us) client_negotiation.cc:769] Sending connection context
0927 02:40:25.204688 (+ 203us) client_negotiation.cc:240] Negotiation successful
0927 02:40:25.205030 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":942,"thread_start_us":136,"threads_started":1}
I20260927 02:40:25.206961 11846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:40:25.189872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60045 (local address 127.0.98.62:41825)
0927 02:40:25.193099 (+ 3227us) server_negotiation.cc:206] Beginning negotiation
0927 02:40:25.193104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:40:25.193120 (+ 16us) server_negotiation.cc:407] Connection header received
0927 02:40:25.193163 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:40:25.193166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:40:25.193215 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:40:25.194726 (+ 1511us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:40:25.194822 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:40:25.195598 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:40:25.197171 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:40:25.197514 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:40:25.197609 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:40:25.200689 (+ 3080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:40:25.200693 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:40:25.200695 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:40:25.200722 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:40:25.202575 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:40:25.203225 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:40:25.203229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:40:25.203231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:40:25.203327 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:40:25.203736 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:40:25.203739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:40:25.203740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:40:25.203977 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:40:25.204072 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 02:40:25.206664 (+ 2592us) server_negotiation.cc:299] Negotiation successful
0927 02:40:25.206795 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3128,"thread_start_us":88,"threads_started":1}
I20260927 02:40:25.208762 11844 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41825
I20260927 02:40:25.209015 11844 heartbeater.cc:464] Registering TS with master...
I20260927 02:40:25.209613 11844 heartbeater.cc:511] Master 127.0.98.62:41825 requested a full tablet report, sending...
I20260927 02:40:25.211421 11630 ts_manager.cc:194] Registered new tserver with Master: b5dffd2d01b04826bbcb2d0f43415b2a (127.0.98.1:37391)
I20260927 02:40:25.212971 11630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60045
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Sep 27 02:40:25 dist-test-slave-zv8z krb5kdc[11154](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790476825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:40:25.245937 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39187/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41825
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:40:25.405999 11850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:40:25.406729 11850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:40:25.406847 11850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:40:25.412438 11850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:40:25.412565 11850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:40:25.412683 11850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:40:25.420102 11850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:39187/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:40:25.421655 11850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:40:25.423142 11850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:40:25.436849 11850 server_base.cc:1034] running on GCE node
W20260927 02:40:25.437424 11859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:40:25.437873 11856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:40:25.442408 11857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:40:25.443087 11850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:40:25.443931 11850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:40:25.451376 11850 hybrid_clock.cc:648] HybridClock initialized: now 1790476825451309 us; error 91 us; skew 500 ppm
Sep 27 02:40:25 dist-test-slave-zv8z krb5kdc[11154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:40:25.455072 11850 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:40:25.456174 11850 webserver.cc:460] Webserver started at http://127.0.98.2:41293/ using document root <none> and password file <none>
I20260927 02:40:25.456884 11850 fs_manager.cc:359] Metadata directory not provided
I20260927 02:40:25.457021 11850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:40:25.457285 11850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 02:40:25 dist-test-slave-zv8z krb5kdc[11154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:40:25.479755 11850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ts-1/data/instance:
uuid: "659af5a3c2cf4b98a9b2af97923e7a49"
format_stamp: "Formatted at 2026-09-27 02:40:25 on dist-test-slave-zv8z"
server_key: "626574cdb4fc74cde7ee21fad7b38c01"
server_key_iv: "e368c81f82ba620ce14d9fa7f3244190"
server_key_version: "kuduclusterkey@0"
I20260927 02:40:25.481616 11850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ts-1/wal/instance:
uuid: "659af5a3c2cf4b98a9b2af97923e7a49"
format_stamp: "Formatted at 2026-09-27 02:40:25 on dist-test-slave-zv8z"
server_key: "626574cdb4fc74cde7ee21fad7b38c01"
server_key_iv: "e368c81f82ba620ce14d9fa7f3244190"
server_key_version: "kuduclusterkey@0"
I20260927 02:40:25.487001 11850 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 02:40:25.506814 11866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:40:25.508491 11850 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260927 02:40:25.508620 11850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ts-1/wal
uuid: "659af5a3c2cf4b98a9b2af97923e7a49"
format_stamp: "Formatted at 2026-09-27 02:40:25 on dist-test-slave-zv8z"
server_key: "626574cdb4fc74cde7ee21fad7b38c01"
server_key_iv: "e368c81f82ba620ce14d9fa7f3244190"
server_key_version: "kuduclusterkey@0"
I20260927 02:40:25.508742 11850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:40:25.551801 11850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:40:25.555327 11850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:40:25.555513 11850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:40:25.556142 11850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:40:25.557133 11850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:40:25.557237 11850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:40:25.557325 11850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:40:25.557381 11850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:40:25.567976 11850 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44695
I20260927 02:40:25.568028 11979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44695 every 8 connection(s)
I20260927 02:40:25.569193 11850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:40:25.578619 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 11850
I20260927 02:40:25.578753 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:40:25.579084 392 external_mini_cluster.cc:1428] Setting key 484f5ee79ed65ee7cdc40bd0fd99a62b
Sep 27 02:40:25 dist-test-slave-zv8z krb5kdc[11154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:40:25.596477 11846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:40:25.571359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53789 (local address 127.0.98.62:41825)
0927 02:40:25.572848 (+ 1489us) server_negotiation.cc:206] Beginning negotiation
0927 02:40:25.572853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:40:25.573313 (+ 460us) server_negotiation.cc:407] Connection header received
0927 02:40:25.583676 (+ 10363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:40:25.583680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:40:25.583740 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:40:25.583836 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:40:25.585572 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:40:25.586441 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:40:25.589073 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:40:25.589421 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:40:25.589535 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:40:25.592443 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:40:25.592446 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:40:25.592449 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:40:25.592478 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:40:25.594115 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:40:25.594741 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:40:25.594745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:40:25.594747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:40:25.594836 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:40:25.595204 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:40:25.595208 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:40:25.595209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:40:25.595428 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:40:25.595502 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 02:40:25.596177 (+ 675us) server_negotiation.cc:299] Negotiation successful
0927 02:40:25.596303 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1381}
I20260927 02:40:25.598284 11982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:40:25.571818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41825 (local address 127.0.98.2:53789)
0927 02:40:25.573121 (+ 1303us) negotiation.cc:107] Waiting for socket to connect
0927 02:40:25.573186 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 02:40:25.574188 (+ 1002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:40:25.584069 (+ 9881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:40:25.584081 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:40:25.584566 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:40:25.585378 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:40:25.585400 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:40:25.587598 (+ 2198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:40:25.587602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:40:25.588957 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:40:25.588968 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:40:25.589604 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:40:25.589608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:40:25.589821 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:40:25.590514 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:40:25.590534 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:40:25.592261 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:40:25.594240 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:40:25.594285 (+ 45us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:40:25.594304 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:40:25.594631 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:40:25.594938 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:40:25.594941 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:40:25.594943 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:40:25.595110 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:40:25.595566 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:40:25.595572 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:40:25.595909 (+ 337us) client_negotiation.cc:769] Sending connection context
0927 02:40:25.596858 (+ 949us) client_negotiation.cc:240] Negotiation successful
0927 02:40:25.597202 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1089,"thread_start_us":133,"threads_started":1}
I20260927 02:40:25.599726 11980 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41825
I20260927 02:40:25.600013 11980 heartbeater.cc:464] Registering TS with master...
I20260927 02:40:25.600618 11980 heartbeater.cc:511] Master 127.0.98.62:41825 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:40:25.601692 11630 ts_manager.cc:194] Registered new tserver with Master: 659af5a3c2cf4b98a9b2af97923e7a49 (127.0.98.2:44695)
I20260927 02:40:25.602443 11630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53789
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:40:25.629803 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39187/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41825
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 02:40:25 dist-test-slave-zv8z krb5kdc[11154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260927 02:40:25.809146 11985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:40:25.809409 11985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:40:25.809468 11985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:40:25.813781 11985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:40:25.813845 11985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:40:25.813927 11985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:40:25.819990 11985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:39187/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:40:25.821167 11985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:40:25.822445 11985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:40:25.840981 11991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:40:25.840981 11990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:40:25.841472 11985 server_base.cc:1034] running on GCE node
W20260927 02:40:25.841593 11993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:40:25.842037 11985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:40:25.842875 11985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:40:25.844105 11985 hybrid_clock.cc:648] HybridClock initialized: now 1790476825844037 us; error 85 us; skew 500 ppm
Sep 27 02:40:25 dist-test-slave-zv8z krb5kdc[11154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:40:25.847829 11985 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:40:25.848941 11985 webserver.cc:460] Webserver started at http://127.0.98.3:34653/ using document root <none> and password file <none>
I20260927 02:40:25.849613 11985 fs_manager.cc:359] Metadata directory not provided
I20260927 02:40:25.849685 11985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:40:25.849916 11985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 02:40:25 dist-test-slave-zv8z krb5kdc[11154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:40:25.886359 11985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ts-2/data/instance:
uuid: "61aa3dc2c1db462598d7964d7840ec2b"
format_stamp: "Formatted at 2026-09-27 02:40:25 on dist-test-slave-zv8z"
server_key: "600d1e4fe363c536179715c485c306a8"
server_key_iv: "f7db73079bf7ecb5d4ba3d3cc5a01785"
server_key_version: "kuduclusterkey@0"
I20260927 02:40:25.891862 11985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ts-2/wal/instance:
uuid: "61aa3dc2c1db462598d7964d7840ec2b"
format_stamp: "Formatted at 2026-09-27 02:40:25 on dist-test-slave-zv8z"
server_key: "600d1e4fe363c536179715c485c306a8"
server_key_iv: "f7db73079bf7ecb5d4ba3d3cc5a01785"
server_key_version: "kuduclusterkey@0"
I20260927 02:40:25.908344 11985 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.004s sys 0.000s
I20260927 02:40:25.941612 12000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:40:25.943320 11985 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 02:40:25.943445 11985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ts-2/wal
uuid: "61aa3dc2c1db462598d7964d7840ec2b"
format_stamp: "Formatted at 2026-09-27 02:40:25 on dist-test-slave-zv8z"
server_key: "600d1e4fe363c536179715c485c306a8"
server_key_iv: "f7db73079bf7ecb5d4ba3d3cc5a01785"
server_key_version: "kuduclusterkey@0"
I20260927 02:40:25.943576 11985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:40:25.997401 11985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:40:26.008167 11985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:40:26.008419 11985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:40:26.009128 11985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:40:26.010229 11985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:40:26.010350 11985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:40:26.010435 11985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:40:26.010489 11985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:40:26.020681 11985 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45815
I20260927 02:40:26.021629 12113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45815 every 8 connection(s)
I20260927 02:40:26.021880 11985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:40:26.025183 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 11985
I20260927 02:40:26.025305 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:40:26.025564 392 external_mini_cluster.cc:1428] Setting key 4a273465c949ef1c3dbd3feeafe92c82
Sep 27 02:40:26 dist-test-slave-zv8z krb5kdc[11154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:40:26.038966 11846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:40:26.023971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50593 (local address 127.0.98.62:41825)
0927 02:40:26.024096 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 02:40:26.024100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:40:26.025794 (+ 1694us) server_negotiation.cc:407] Connection header received
0927 02:40:26.027080 (+ 1286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:40:26.027084 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:40:26.027138 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:40:26.027188 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:40:26.028758 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:40:26.029581 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:40:26.031282 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:40:26.031644 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:40:26.031722 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:40:26.034666 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:40:26.034671 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:40:26.034674 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:40:26.034702 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:40:26.036380 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:40:26.037082 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:40:26.037086 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:40:26.037088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:40:26.037185 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:40:26.037605 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:40:26.037609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:40:26.037611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:40:26.037847 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:40:26.037940 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 02:40:26.038664 (+ 724us) server_negotiation.cc:299] Negotiation successful
0927 02:40:26.038797 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 02:40:26.040894 12116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:40:26.024431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41825 (local address 127.0.98.3:50593)
0927 02:40:26.025545 (+ 1114us) negotiation.cc:107] Waiting for socket to connect
0927 02:40:26.025648 (+ 103us) client_negotiation.cc:174] Beginning negotiation
0927 02:40:26.026835 (+ 1187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:40:26.027601 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:40:26.027610 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:40:26.027944 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:40:26.028588 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:40:26.028609 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:40:26.029723 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:40:26.029727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:40:26.031143 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:40:26.031155 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:40:26.031921 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:40:26.031925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:40:26.032120 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:40:26.032807 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:40:26.032828 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:40:26.034469 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:40:26.036486 (+ 2017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:40:26.036492 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:40:26.036508 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:40:26.036951 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:40:26.037277 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:40:26.037280 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:40:26.037282 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:40:26.037464 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:40:26.037996 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:40:26.038002 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:40:26.038355 (+ 353us) client_negotiation.cc:769] Sending connection context
0927 02:40:26.039450 (+ 1095us) client_negotiation.cc:240] Negotiation successful
0927 02:40:26.039812 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":150,"threads_started":1}
I20260927 02:40:26.041958 12114 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41825
I20260927 02:40:26.042217 12114 heartbeater.cc:464] Registering TS with master...
I20260927 02:40:26.042785 12114 heartbeater.cc:511] Master 127.0.98.62:41825 requested a full tablet report, sending...
I20260927 02:40:26.043882 11630 ts_manager.cc:194] Registered new tserver with Master: 61aa3dc2c1db462598d7964d7840ec2b (127.0.98.3:45815)
I20260927 02:40:26.044602 11630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50593
I20260927 02:40:26.050992 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:40:26.062451 11846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:40:26.053309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34248 (local address 127.0.98.62:41825)
0927 02:40:26.053425 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 02:40:26.053429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:40:26.053666 (+ 237us) server_negotiation.cc:407] Connection header received
0927 02:40:26.053782 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:40:26.053786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:40:26.053834 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:40:26.053915 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:40:26.054705 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:40:26.055473 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:40:26.056611 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:40:26.056914 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:40:26.056978 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:40:26.058904 (+ 1926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:40:26.058908 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:40:26.058911 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:40:26.058936 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:40:26.060597 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:40:26.061066 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:40:26.061070 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:40:26.061072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:40:26.061168 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:40:26.061481 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:40:26.061484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:40:26.061486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:40:26.061697 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:40:26.061799 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 02:40:26.062154 (+ 355us) server_negotiation.cc:299] Negotiation successful
0927 02:40:26.062297 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 02:40:26.066306 11630 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34248:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:40:26.070904 11630 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Sep 27 02:40:26 dist-test-slave-zv8z krb5kdc[11154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:40:26.096552 12131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:40:26.088233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37391 (local address 127.0.0.1:51740)
0927 02:40:26.090746 (+ 2513us) negotiation.cc:107] Waiting for socket to connect
0927 02:40:26.090758 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:40:26.090846 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:40:26.091520 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:40:26.091526 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:40:26.091549 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:40:26.091700 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:40:26.091708 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:40:26.093833 (+ 2125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:40:26.093836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:40:26.095008 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:40:26.095019 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:40:26.096036 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:40:26.096040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:40:26.096158 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:40:26.096194 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 02:40:26.096296 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 02:40:26.096407 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2459,"thread_start_us":99,"threads_started":1}
I20260927 02:40:26.097103 12129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:40:26.088175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44695 (local address 127.0.0.1:44934)
0927 02:40:26.090167 (+ 1992us) negotiation.cc:107] Waiting for socket to connect
0927 02:40:26.090220 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 02:40:26.090376 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:40:26.091609 (+ 1233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:40:26.091613 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:40:26.091630 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:40:26.091717 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:40:26.091723 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:40:26.093056 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:40:26.093059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:40:26.094446 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:40:26.094456 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:40:26.096574 (+ 2118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:40:26.096578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:40:26.096854 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:40:26.096876 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 02:40:26.096925 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 02:40:26.096985 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1810,"thread_start_us":104,"threads_started":1}
I20260927 02:40:26.097600 12128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:40:26.088272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44934 (local address 127.0.98.2:44695)
0927 02:40:26.091046 (+ 2774us) server_negotiation.cc:206] Beginning negotiation
0927 02:40:26.091061 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 02:40:26.091081 (+ 20us) server_negotiation.cc:407] Connection header received
0927 02:40:26.091157 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:40:26.091162 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:40:26.091350 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:40:26.091480 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:40:26.091827 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:40:26.092938 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:40:26.094591 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:40:26.095700 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:40:26.095792 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:40:26.097339 (+ 1547us) server_negotiation.cc:1035] Waiting for connection context
0927 02:40:26.097388 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 02:40:26.097475 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2681,"thread_start_us":65,"threads_started":1}
I20260927 02:40:26.098050 12130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:40:26.088679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51740 (local address 127.0.98.1:37391)
0927 02:40:26.090626 (+ 1947us) server_negotiation.cc:206] Beginning negotiation
0927 02:40:26.090634 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:40:26.090863 (+ 229us) server_negotiation.cc:407] Connection header received
0927 02:40:26.090960 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:40:26.090966 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:40:26.091204 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:40:26.091352 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:40:26.091831 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:40:26.093708 (+ 1877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:40:26.095157 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:40:26.095941 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:40:26.096036 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:40:26.096185 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 02:40:26.096323 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 02:40:26.096413 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1849,"thread_start_us":59,"threads_started":1}
I20260927 02:40:26.100684 11898 tablet_service.cc:1467] Processing CreateTablet for tablet bc2c4de9a1e14afb9a69ca7795ec0256 (DEFAULT_TABLE table=test-table [id=26b70aefc28f4ea5aae25a27d3ef4bc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:40:26.101836 11898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc2c4de9a1e14afb9a69ca7795ec0256. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:40:26.101876 11778 tablet_service.cc:1467] Processing CreateTablet for tablet bc2c4de9a1e14afb9a69ca7795ec0256 (DEFAULT_TABLE table=test-table [id=26b70aefc28f4ea5aae25a27d3ef4bc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:40:26.102530 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:40:26.089305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45815 (local address 127.0.0.1:54162)
0927 02:40:26.095213 (+ 5908us) negotiation.cc:107] Waiting for socket to connect
0927 02:40:26.095226 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:40:26.095307 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:40:26.098371 (+ 3064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:40:26.098374 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:40:26.098392 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:40:26.098475 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:40:26.098481 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:40:26.099888 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:40:26.099891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:40:26.101052 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:40:26.101063 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:40:26.102145 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:40:26.102149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:40:26.102241 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:40:26.102288 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 02:40:26.102338 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 02:40:26.102402 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5848,"thread_start_us":79,"threads_started":1}
I20260927 02:40:26.102828 11778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc2c4de9a1e14afb9a69ca7795ec0256. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:40:26.103904 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:40:26.089697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54162 (local address 127.0.98.3:45815)
0927 02:40:26.097893 (+ 8196us) server_negotiation.cc:206] Beginning negotiation
0927 02:40:26.097904 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:40:26.097927 (+ 23us) server_negotiation.cc:407] Connection header received
0927 02:40:26.097995 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:40:26.098000 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:40:26.098206 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:40:26.098623 (+ 417us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:40:26.098757 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:40:26.099769 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:40:26.101222 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:40:26.101996 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:40:26.103493 (+ 1497us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:40:26.103629 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 02:40:26.103676 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 02:40:26.103761 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8093,"thread_start_us":140,"threads_started":1}
I20260927 02:40:26.106745 12048 tablet_service.cc:1467] Processing CreateTablet for tablet bc2c4de9a1e14afb9a69ca7795ec0256 (DEFAULT_TABLE table=test-table [id=26b70aefc28f4ea5aae25a27d3ef4bc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:40:26.107676 12048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc2c4de9a1e14afb9a69ca7795ec0256. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:40:26.117805 12134 tablet_bootstrap.cc:492] T bc2c4de9a1e14afb9a69ca7795ec0256 P 61aa3dc2c1db462598d7964d7840ec2b: Bootstrap starting.
I20260927 02:40:26.118850 12133 tablet_bootstrap.cc:492] T bc2c4de9a1e14afb9a69ca7795ec0256 P b5dffd2d01b04826bbcb2d0f43415b2a: Bootstrap starting.
I20260927 02:40:26.120564 12134 tablet_bootstrap.cc:654] T bc2c4de9a1e14afb9a69ca7795ec0256 P 61aa3dc2c1db462598d7964d7840ec2b: Neither blocks nor log segments found. Creating new log.
I20260927 02:40:26.120975 12133 tablet_bootstrap.cc:654] T bc2c4de9a1e14afb9a69ca7795ec0256 P b5dffd2d01b04826bbcb2d0f43415b2a: Neither blocks nor log segments found. Creating new log.
I20260927 02:40:26.121438 12134 log.cc:824] T bc2c4de9a1e14afb9a69ca7795ec0256 P 61aa3dc2c1db462598d7964d7840ec2b: Log is configured to *not* fsync() on all Append() calls
I20260927 02:40:26.121757 12133 log.cc:824] T bc2c4de9a1e14afb9a69ca7795ec0256 P b5dffd2d01b04826bbcb2d0f43415b2a: Log is configured to *not* fsync() on all Append() calls
I20260927 02:40:26.124301 12133 tablet_bootstrap.cc:492] T bc2c4de9a1e14afb9a69ca7795ec0256 P b5dffd2d01b04826bbcb2d0f43415b2a: No bootstrap required, opened a new log
I20260927 02:40:26.124500 12134 tablet_bootstrap.cc:492] T bc2c4de9a1e14afb9a69ca7795ec0256 P 61aa3dc2c1db462598d7964d7840ec2b: No bootstrap required, opened a new log
I20260927 02:40:26.124524 12133 ts_tablet_manager.cc:1397] T bc2c4de9a1e14afb9a69ca7795ec0256 P b5dffd2d01b04826bbcb2d0f43415b2a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 02:40:26.124699 12134 ts_tablet_manager.cc:1397] T bc2c4de9a1e14afb9a69ca7795ec0256 P 61aa3dc2c1db462598d7964d7840ec2b: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 02:40:26.128748 12133 raft_consensus.cc:348] T bc2c4de9a1e14afb9a69ca7795ec0256 P b5dffd2d01b04826bbcb2d0f43415b2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659af5a3c2cf4b98a9b2af97923e7a49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44695 } } peers { permanent_uuid: "b5dffd2d01b04826bbcb2d0f43415b2a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37391 } } peers { permanent_uuid: "61aa3dc2c1db462598d7964d7840ec2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45815 } }
I20260927 02:40:26.129065 12133 raft_consensus.cc:374] T bc2c4de9a1e14afb9a69ca7795ec0256 P b5dffd2d01b04826bbcb2d0f43415b2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:40:26.128939 12134 raft_consensus.cc:348] T bc2c4de9a1e14afb9a69ca7795ec0256 P 61aa3dc2c1db462598d7964d7840ec2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659af5a3c2cf4b98a9b2af97923e7a49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44695 } } peers { permanent_uuid: "b5dffd2d01b04826bbcb2d0f43415b2a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37391 } } peers { permanent_uuid: "61aa3dc2c1db462598d7964d7840ec2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45815 } }
I20260927 02:40:26.129153 12133 raft_consensus.cc:729] T bc2c4de9a1e14afb9a69ca7795ec0256 P b5dffd2d01b04826bbcb2d0f43415b2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5dffd2d01b04826bbcb2d0f43415b2a, State: Initialized, Role: FOLLOWER
I20260927 02:40:26.129266 12134 raft_consensus.cc:374] T bc2c4de9a1e14afb9a69ca7795ec0256 P 61aa3dc2c1db462598d7964d7840ec2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:40:26.129361 12134 raft_consensus.cc:729] T bc2c4de9a1e14afb9a69ca7795ec0256 P 61aa3dc2c1db462598d7964d7840ec2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61aa3dc2c1db462598d7964d7840ec2b, State: Initialized, Role: FOLLOWER
I20260927 02:40:26.129681 12133 consensus_queue.cc:260] T bc2c4de9a1e14afb9a69ca7795ec0256 P b5dffd2d01b04826bbcb2d0f43415b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659af5a3c2cf4b98a9b2af97923e7a49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44695 } } peers { permanent_uuid: "b5dffd2d01b04826bbcb2d0f43415b2a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37391 } } peers { permanent_uuid: "61aa3dc2c1db462598d7964d7840ec2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45815 } }
I20260927 02:40:26.129786 12134 consensus_queue.cc:260] T bc2c4de9a1e14afb9a69ca7795ec0256 P 61aa3dc2c1db462598d7964d7840ec2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659af5a3c2cf4b98a9b2af97923e7a49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44695 } } peers { permanent_uuid: "b5dffd2d01b04826bbcb2d0f43415b2a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37391 } } peers { permanent_uuid: "61aa3dc2c1db462598d7964d7840ec2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45815 } }
I20260927 02:40:26.130795 12133 ts_tablet_manager.cc:1428] T bc2c4de9a1e14afb9a69ca7795ec0256 P b5dffd2d01b04826bbcb2d0f43415b2a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 02:40:26.130795 12134 ts_tablet_manager.cc:1428] T bc2c4de9a1e14afb9a69ca7795ec0256 P 61aa3dc2c1db462598d7964d7840ec2b: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20260927 02:40:26.131273 11844 heartbeater.cc:503] Master 127.0.98.62:41825 was elected leader, sending a full tablet report...
I20260927 02:40:26.134159 12114 heartbeater.cc:503] Master 127.0.98.62:41825 was elected leader, sending a full tablet report...
I20260927 02:40:26.138921 12137 tablet_bootstrap.cc:492] T bc2c4de9a1e14afb9a69ca7795ec0256 P 659af5a3c2cf4b98a9b2af97923e7a49: Bootstrap starting.
I20260927 02:40:26.141518 12137 tablet_bootstrap.cc:654] T bc2c4de9a1e14afb9a69ca7795ec0256 P 659af5a3c2cf4b98a9b2af97923e7a49: Neither blocks nor log segments found. Creating new log.
I20260927 02:40:26.142506 12137 log.cc:824] T bc2c4de9a1e14afb9a69ca7795ec0256 P 659af5a3c2cf4b98a9b2af97923e7a49: Log is configured to *not* fsync() on all Append() calls
I20260927 02:40:26.152431 12137 tablet_bootstrap.cc:492] T bc2c4de9a1e14afb9a69ca7795ec0256 P 659af5a3c2cf4b98a9b2af97923e7a49: No bootstrap required, opened a new log
I20260927 02:40:26.152791 12137 ts_tablet_manager.cc:1397] T bc2c4de9a1e14afb9a69ca7795ec0256 P 659af5a3c2cf4b98a9b2af97923e7a49: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20260927 02:40:26.157421 12137 raft_consensus.cc:348] T bc2c4de9a1e14afb9a69ca7795ec0256 P 659af5a3c2cf4b98a9b2af97923e7a49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659af5a3c2cf4b98a9b2af97923e7a49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44695 } } peers { permanent_uuid: "b5dffd2d01b04826bbcb2d0f43415b2a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37391 } } peers { permanent_uuid: "61aa3dc2c1db462598d7964d7840ec2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45815 } }
I20260927 02:40:26.157820 12137 raft_consensus.cc:374] T bc2c4de9a1e14afb9a69ca7795ec0256 P 659af5a3c2cf4b98a9b2af97923e7a49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:40:26.157924 12137 raft_consensus.cc:729] T bc2c4de9a1e14afb9a69ca7795ec0256 P 659af5a3c2cf4b98a9b2af97923e7a49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 659af5a3c2cf4b98a9b2af97923e7a49, State: Initialized, Role: FOLLOWER
I20260927 02:40:26.158541 12137 consensus_queue.cc:260] T bc2c4de9a1e14afb9a69ca7795ec0256 P 659af5a3c2cf4b98a9b2af97923e7a49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659af5a3c2cf4b98a9b2af97923e7a49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44695 } } peers { permanent_uuid: "b5dffd2d01b04826bbcb2d0f43415b2a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37391 } } peers { permanent_uuid: "61aa3dc2c1db462598d7964d7840ec2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45815 } }
I20260927 02:40:26.159789 12137 ts_tablet_manager.cc:1428] T bc2c4de9a1e14afb9a69ca7795ec0256 P 659af5a3c2cf4b98a9b2af97923e7a49: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 02:40:26.160257 11980 heartbeater.cc:503] Master 127.0.98.62:41825 was elected leader, sending a full tablet report...
W20260927 02:40:26.194397 11845 tablet.cc:2369] T bc2c4de9a1e14afb9a69ca7795ec0256 P b5dffd2d01b04826bbcb2d0f43415b2a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:40:26.215085 12138 raft_consensus.cc:482] T bc2c4de9a1e14afb9a69ca7795ec0256 P b5dffd2d01b04826bbcb2d0f43415b2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:40:26.215324 12138 raft_consensus.cc:504] T bc2c4de9a1e14afb9a69ca7795ec0256 P b5dffd2d01b04826bbcb2d0f43415b2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659af5a3c2cf4b98a9b2af97923e7a49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44695 } } peers { permanent_uuid: "b5dffd2d01b04826bbcb2d0f43415b2a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37391 } } peers { permanent_uuid: "61aa3dc2c1db462598d7964d7840ec2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45815 } }
I20260927 02:40:26.216771 12138 leader_election.cc:288] T bc2c4de9a1e14afb9a69ca7795ec0256 P b5dffd2d01b04826bbcb2d0f43415b2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 659af5a3c2cf4b98a9b2af97923e7a49 (127.0.98.2:44695), 61aa3dc2c1db462598d7964d7840ec2b (127.0.98.3:45815)
I20260927 02:40:26.237205 12128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:40:26.218400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52579 (local address 127.0.98.2:44695)
0927 02:40:26.218536 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 02:40:26.218540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:40:26.229075 (+ 10535us) server_negotiation.cc:407] Connection header received
0927 02:40:26.229312 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:40:26.229316 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:40:26.229371 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:40:26.229420 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:40:26.229857 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:40:26.230645 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:40:26.235585 (+ 4940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:40:26.236216 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:40:26.236300 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:40:26.236370 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 02:40:26.236955 (+ 585us) server_negotiation.cc:299] Negotiation successful
0927 02:40:26.237046 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 02:40:26.238577 12144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:40:26.217199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44695 (local address 127.0.98.1:52579)
0927 02:40:26.228944 (+ 11745us) negotiation.cc:107] Waiting for socket to connect
0927 02:40:26.228965 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 02:40:26.229178 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:40:26.229584 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:40:26.229587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:40:26.229607 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:40:26.229715 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:40:26.229721 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:40:26.230793 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:40:26.230796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:40:26.233375 (+ 2579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:40:26.233386 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:40:26.236463 (+ 3077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:40:26.236466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:40:26.236568 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:40:26.236592 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 02:40:26.237764 (+ 1172us) client_negotiation.cc:240] Negotiation successful
0927 02:40:26.237848 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11639,"thread_start_us":110,"threads_started":1}
I20260927 02:40:26.240265 11916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc2c4de9a1e14afb9a69ca7795ec0256" candidate_uuid: "b5dffd2d01b04826bbcb2d0f43415b2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "659af5a3c2cf4b98a9b2af97923e7a49" is_pre_election: true
I20260927 02:40:26.240628 11916 raft_consensus.cc:2454] T bc2c4de9a1e14afb9a69ca7795ec0256 P 659af5a3c2cf4b98a9b2af97923e7a49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5dffd2d01b04826bbcb2d0f43415b2a in term 0.
I20260927 02:40:26.241252 11731 leader_election.cc:302] T bc2c4de9a1e14afb9a69ca7795ec0256 P b5dffd2d01b04826bbcb2d0f43415b2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 659af5a3c2cf4b98a9b2af97923e7a49, b5dffd2d01b04826bbcb2d0f43415b2a; no voters:
I20260927 02:40:26.241586 12138 raft_consensus.cc:2781] T bc2c4de9a1e14afb9a69ca7795ec0256 P b5dffd2d01b04826bbcb2d0f43415b2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:40:26.241662 12138 raft_consensus.cc:482] T bc2c4de9a1e14afb9a69ca7795ec0256 P b5dffd2d01b04826bbcb2d0f43415b2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:40:26.241700 12138 raft_consensus.cc:3037] T bc2c4de9a1e14afb9a69ca7795ec0256 P b5dffd2d01b04826bbcb2d0f43415b2a [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:40:26.243044 12138 raft_consensus.cc:504] T bc2c4de9a1e14afb9a69ca7795ec0256 P b5dffd2d01b04826bbcb2d0f43415b2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659af5a3c2cf4b98a9b2af97923e7a49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44695 } } peers { permanent_uuid: "b5dffd2d01b04826bbcb2d0f43415b2a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37391 } } peers { permanent_uuid: "61aa3dc2c1db462598d7964d7840ec2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45815 } }
I20260927 02:40:26.243396 12138 leader_election.cc:288] T bc2c4de9a1e14afb9a69ca7795ec0256 P b5dffd2d01b04826bbcb2d0f43415b2a [CANDIDATE]: Term 1 election: Requested vote from peers 659af5a3c2cf4b98a9b2af97923e7a49 (127.0.98.2:44695), 61aa3dc2c1db462598d7964d7840ec2b (127.0.98.3:45815)
I20260927 02:40:26.243968 11916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc2c4de9a1e14afb9a69ca7795ec0256" candidate_uuid: "b5dffd2d01b04826bbcb2d0f43415b2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "659af5a3c2cf4b98a9b2af97923e7a49"
I20260927 02:40:26.244086 11916 raft_consensus.cc:3037] T bc2c4de9a1e14afb9a69ca7795ec0256 P 659af5a3c2cf4b98a9b2af97923e7a49 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:40:26.245335 11916 raft_consensus.cc:2454] T bc2c4de9a1e14afb9a69ca7795ec0256 P 659af5a3c2cf4b98a9b2af97923e7a49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5dffd2d01b04826bbcb2d0f43415b2a in term 1.
I20260927 02:40:26.245710 11731 leader_election.cc:302] T bc2c4de9a1e14afb9a69ca7795ec0256 P b5dffd2d01b04826bbcb2d0f43415b2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 659af5a3c2cf4b98a9b2af97923e7a49, b5dffd2d01b04826bbcb2d0f43415b2a; no voters:
I20260927 02:40:26.245921 12138 raft_consensus.cc:2781] T bc2c4de9a1e14afb9a69ca7795ec0256 P b5dffd2d01b04826bbcb2d0f43415b2a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:40:26.246125 12138 raft_consensus.cc:686] T bc2c4de9a1e14afb9a69ca7795ec0256 P b5dffd2d01b04826bbcb2d0f43415b2a [term 1 LEADER]: Becoming Leader. State: Replica: b5dffd2d01b04826bbcb2d0f43415b2a, State: Running, Role: LEADER
I20260927 02:40:26.246532 12138 consensus_queue.cc:237] T bc2c4de9a1e14afb9a69ca7795ec0256 P b5dffd2d01b04826bbcb2d0f43415b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659af5a3c2cf4b98a9b2af97923e7a49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44695 } } peers { permanent_uuid: "b5dffd2d01b04826bbcb2d0f43415b2a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37391 } } peers { permanent_uuid: "61aa3dc2c1db462598d7964d7840ec2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45815 } }
I20260927 02:40:26.250051 11629 catalog_manager.cc:5495] T bc2c4de9a1e14afb9a69ca7795ec0256 P b5dffd2d01b04826bbcb2d0f43415b2a reported cstate change: term changed from 0 to 1, leader changed from <none> to b5dffd2d01b04826bbcb2d0f43415b2a (127.0.98.1). New cstate: current_term: 1 leader_uuid: "b5dffd2d01b04826bbcb2d0f43415b2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659af5a3c2cf4b98a9b2af97923e7a49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5dffd2d01b04826bbcb2d0f43415b2a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61aa3dc2c1db462598d7964d7840ec2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45815 } health_report { overall_health: UNKNOWN } } }
I20260927 02:40:26.253284 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:40:26.218131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46653 (local address 127.0.98.3:45815)
0927 02:40:26.218276 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 02:40:26.218281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:40:26.231671 (+ 13390us) server_negotiation.cc:407] Connection header received
0927 02:40:26.232266 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:40:26.232270 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:40:26.232323 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:40:26.232372 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:40:26.233535 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:40:26.234273 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:40:26.238851 (+ 4578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:40:26.239459 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:40:26.239556 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:40:26.239618 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 02:40:26.253093 (+ 13475us) server_negotiation.cc:299] Negotiation successful
0927 02:40:26.253187 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 02:40:26.253659 12143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:40:26.217584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45815 (local address 127.0.98.1:46653)
0927 02:40:26.231577 (+ 13993us) negotiation.cc:107] Waiting for socket to connect
0927 02:40:26.231592 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:40:26.232125 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:40:26.232496 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:40:26.232499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:40:26.232514 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:40:26.232706 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:40:26.232713 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:40:26.234426 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:40:26.234429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:40:26.235429 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:40:26.235439 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:40:26.251362 (+ 15923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:40:26.251366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:40:26.251492 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:40:26.251532 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 02:40:26.253459 (+ 1927us) client_negotiation.cc:240] Negotiation successful
0927 02:40:26.253553 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13906,"thread_start_us":89,"threads_started":1}
I20260927 02:40:26.254210 12068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc2c4de9a1e14afb9a69ca7795ec0256" candidate_uuid: "b5dffd2d01b04826bbcb2d0f43415b2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61aa3dc2c1db462598d7964d7840ec2b" is_pre_election: true
I20260927 02:40:26.254529 12068 raft_consensus.cc:2454] T bc2c4de9a1e14afb9a69ca7795ec0256 P 61aa3dc2c1db462598d7964d7840ec2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5dffd2d01b04826bbcb2d0f43415b2a in term 0.
I20260927 02:40:26.254841 12067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc2c4de9a1e14afb9a69ca7795ec0256" candidate_uuid: "b5dffd2d01b04826bbcb2d0f43415b2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61aa3dc2c1db462598d7964d7840ec2b"
I20260927 02:40:26.254942 12067 raft_consensus.cc:3037] T bc2c4de9a1e14afb9a69ca7795ec0256 P 61aa3dc2c1db462598d7964d7840ec2b [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:40:26.256165 12067 raft_consensus.cc:2454] T bc2c4de9a1e14afb9a69ca7795ec0256 P 61aa3dc2c1db462598d7964d7840ec2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5dffd2d01b04826bbcb2d0f43415b2a in term 1.
W20260927 02:40:26.274098 12115 tablet.cc:2369] T bc2c4de9a1e14afb9a69ca7795ec0256 P 61aa3dc2c1db462598d7964d7840ec2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:40:26.287079 12130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:40:26.281787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51754 (local address 127.0.98.1:37391)
0927 02:40:26.281964 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 02:40:26.281969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:40:26.281983 (+ 14us) server_negotiation.cc:407] Connection header received
0927 02:40:26.282081 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:40:26.282084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:40:26.282151 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:40:26.282279 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:40:26.282662 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:40:26.283433 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:40:26.284783 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:40:26.285147 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:40:26.285241 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:40:26.285644 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:40:26.286274 (+ 630us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:40:26.286389 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 02:40:26.286842 (+ 453us) server_negotiation.cc:299] Negotiation successful
0927 02:40:26.286905 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260927 02:40:26.301132 12067 raft_consensus.cc:1261] T bc2c4de9a1e14afb9a69ca7795ec0256 P 61aa3dc2c1db462598d7964d7840ec2b [term 1 FOLLOWER]: Refusing update from remote peer b5dffd2d01b04826bbcb2d0f43415b2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:40:26.302328 12145 consensus_queue.cc:1035] T bc2c4de9a1e14afb9a69ca7795ec0256 P b5dffd2d01b04826bbcb2d0f43415b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "61aa3dc2c1db462598d7964d7840ec2b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:40:26.309322 11916 raft_consensus.cc:1261] T bc2c4de9a1e14afb9a69ca7795ec0256 P 659af5a3c2cf4b98a9b2af97923e7a49 [term 1 FOLLOWER]: Refusing update from remote peer b5dffd2d01b04826bbcb2d0f43415b2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:40:26.310053 12145 consensus_queue.cc:1035] T bc2c4de9a1e14afb9a69ca7795ec0256 P b5dffd2d01b04826bbcb2d0f43415b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "659af5a3c2cf4b98a9b2af97923e7a49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 02:40:26.321182 11981 tablet.cc:2369] T bc2c4de9a1e14afb9a69ca7795ec0256 P 659af5a3c2cf4b98a9b2af97923e7a49: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:40:26.326016 12148 mvcc.cc:205] Tried to move back new op lower bound from 7333793080520458240 to 7333793080309596160. Current Snapshot: MvccSnapshot[applied={T|T < 7333793080520458240}]
I20260927 02:40:26.333799 12151 mvcc.cc:205] Tried to move back new op lower bound from 7333793080520458240 to 7333793080309596160. Current Snapshot: MvccSnapshot[applied={T|T < 7333793080520458240}]
I20260927 02:40:26.336817 12149 mvcc.cc:205] Tried to move back new op lower bound from 7333793080520458240 to 7333793080309596160. Current Snapshot: MvccSnapshot[applied={T|T < 7333793080520458240}]
I20260927 02:40:26.347967 11629 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:40:26.348160 11629 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:40:26.350881 11629 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e0060233a8534cb4909fd0dcab01de84: Sending DeleteTablet for 3 replicas of tablet bc2c4de9a1e14afb9a69ca7795ec0256
I20260927 02:40:26.352118 11898 tablet_service.cc:1514] Processing DeleteTablet for tablet bc2c4de9a1e14afb9a69ca7795ec0256 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:40:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44934
I20260927 02:40:26.352283 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 11711
I20260927 02:40:26.352615 12048 tablet_service.cc:1514] Processing DeleteTablet for tablet bc2c4de9a1e14afb9a69ca7795ec0256 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:40:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54162
I20260927 02:40:26.354734 12158 tablet_replica.cc:330] T bc2c4de9a1e14afb9a69ca7795ec0256 P 61aa3dc2c1db462598d7964d7840ec2b: stopping tablet replica
I20260927 02:40:26.355105 12158 raft_consensus.cc:2229] T bc2c4de9a1e14afb9a69ca7795ec0256 P 61aa3dc2c1db462598d7964d7840ec2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:40:26.355460 12158 raft_consensus.cc:2258] T bc2c4de9a1e14afb9a69ca7795ec0256 P 61aa3dc2c1db462598d7964d7840ec2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:40:26.355858 12157 tablet_replica.cc:330] T bc2c4de9a1e14afb9a69ca7795ec0256 P 659af5a3c2cf4b98a9b2af97923e7a49: stopping tablet replica
I20260927 02:40:26.356190 12157 raft_consensus.cc:2229] T bc2c4de9a1e14afb9a69ca7795ec0256 P 659af5a3c2cf4b98a9b2af97923e7a49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:40:26.356509 12157 raft_consensus.cc:2258] T bc2c4de9a1e14afb9a69ca7795ec0256 P 659af5a3c2cf4b98a9b2af97923e7a49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:40:26.358139 12158 ts_tablet_manager.cc:1905] T bc2c4de9a1e14afb9a69ca7795ec0256 P 61aa3dc2c1db462598d7964d7840ec2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:40:26.358201 12157 ts_tablet_manager.cc:1905] T bc2c4de9a1e14afb9a69ca7795ec0256 P 659af5a3c2cf4b98a9b2af97923e7a49: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 02:40:26.359783 11616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37391 (error 108)
I20260927 02:40:26.360682 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 11850
I20260927 02:40:26.362007 12158 ts_tablet_manager.cc:1918] T bc2c4de9a1e14afb9a69ca7795ec0256 P 61aa3dc2c1db462598d7964d7840ec2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:40:26.362442 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:40:26.362113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37391 (local address 127.0.0.1:51762)
0927 02:40:26.362273 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 02:40:26.362351 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 02:40:26.362628 11616 catalog_manager.cc:4584] TS b5dffd2d01b04826bbcb2d0f43415b2a (127.0.98.1:37391): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bc2c4de9a1e14afb9a69ca7795ec0256: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37391: connect: Connection refused (error 111)
I20260927 02:40:26.362874 12158 log.cc:1196] T bc2c4de9a1e14afb9a69ca7795ec0256 P 61aa3dc2c1db462598d7964d7840ec2b: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790476718023131-392-0/minicluster-data/ts-2/wal/wals/bc2c4de9a1e14afb9a69ca7795ec0256
I20260927 02:40:26.363528 12158 ts_tablet_manager.cc:1939] T bc2c4de9a1e14afb9a69ca7795ec0256 P 61aa3dc2c1db462598d7964d7840ec2b: Deleting consensus metadata
I20260927 02:40:26.364849 11615 catalog_manager.cc:4841] TS 61aa3dc2c1db462598d7964d7840ec2b (127.0.98.3:45815): tablet bc2c4de9a1e14afb9a69ca7795ec0256 (table test-table [id=26b70aefc28f4ea5aae25a27d3ef4bc9]) successfully deleted
I20260927 02:40:26.366706 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:40:26.366481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44695 (local address 127.0.0.1:44942)
0927 02:40:26.366590 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 02:40:26.366636 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 02:40:26.366845 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 11985
I20260927 02:40:26.371699 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 11598
I20260927 02:40:26.376832 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 02:40:26.711196 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 02:40:26.711309 392 mini_ranger.cc:66] Stopping Ranger...
I20260927 02:40:27.045509 392 mini_ranger.cc:68] Stopped Ranger
2026-09-27 02:40:27.045 UTC [11184] LOG: received smart shutdown request
2026-09-27 02:40:27.048 UTC [11184] LOG: background worker "logical replication launcher" (PID 11193) exited with exit code 1
2026-09-27 02:40:27.048 UTC [11188] LOG: shutting down
2026-09-27 02:40:27.213 UTC [11184] LOG: database system is shut down
2026-09-27T02:40:27Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46130 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:40:27 dist-test-slave-zv8z krb5kdc[12168](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:40:27 dist-test-slave-zv8z krb5kdc[12168](info): set up 2 sockets
Sep 27 02:40:27 dist-test-slave-zv8z krb5kdc[12168](info): commencing operation
krb5kdc: starting...
W20260927 02:40:28.258005 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/test-user.keytab.
Sep 27 02:40:28 dist-test-slave-zv8z krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:40:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:40:28Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260927 02:40:28.412425 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskBFiRWZ/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/postgres -l logfile start
2026-09-27 02:40:30.838 UTC [12198] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 02:40:30.838 UTC [12198] LOG: listening on IPv4 address "127.0.98.20", port 53069
2026-09-27 02:40:30.842 UTC [12198] LOG: listening on Unix socket "/tmp/.s.PGSQL.53069"
2026-09-27 02:40:30.862 UTC [12201] LOG: database system was shut down at 2026-09-27 02:40:29 UTC
2026-09-27 02:40:30.865 UTC [12198] LOG: database system is ready to accept connections
I20260927 02:40:31.729893 392 mini_postgres.cc:96] Postgres bound to 53069
2026-09-27 02:40:31.735 UTC [12209] FATAL: database "slave" does not exist
127.0.98.20:53069 - accepting connections
I20260927 02:40:31.736052 392 mini_ranger.cc:161] Starting Ranger...
I20260927 02:40:31.749886 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 02:40:32.192811 392 mini_ranger.cc:87] Created ranger Postgres database
I20260927 02:40:32.192898 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ranger-admin
2026-09-27 02:40:32,459 [I] DB FLAVOR :POSTGRES
2026-09-27 02:40:32,459 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 02:40:32,459 [I] Checking connection..
2026-09-27 02:40:32,460 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBFiRWZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBFiRWZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 02:40:32,696 [I] Checking connection passed.
2026-09-27 02:40:32,697 [I] --------- Verifying version history table ---------
2026-09-27 02:40:32,697 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBFiRWZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBFiRWZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 02:40:32,940 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 02:40:32,941 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBFiRWZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBFiRWZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 02:40:33,178 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 02:40:33,178 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 02:40:33,179 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBFiRWZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBFiRWZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53069/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBFiRWZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 02:40:33.402 UTC [12273] WARNING: there is no transaction in progress
2026-09-27 02:40:33,407 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 02:40:33,407 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBFiRWZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBFiRWZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 02:40:33,665 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 02:40:33,666 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 02:40:33,666 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBFiRWZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBFiRWZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 02:40:33,915 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBFiRWZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBFiRWZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 02:40:34,144 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBFiRWZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBFiRWZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-zv8z.c.gcp-upstream.internal','N') ;"
2026-09-27 02:40:34,391 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBFiRWZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBFiRWZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 02:40:34,630 [I] Table x_portal_user does not exist in database ranger
2026-09-27 02:40:34,630 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBFiRWZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBFiRWZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 02:40:34,864 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 02:40:34,864 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBFiRWZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBFiRWZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 02:40:35,101 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBFiRWZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBFiRWZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 02:40:35,330 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 02:40:35,330 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBFiRWZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBFiRWZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53069/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBFiRWZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 02:40:36.245 UTC [12381] WARNING: there is no transaction in progress
2026-09-27 02:40:36.260 UTC [12381] WARNING: there is no transaction in progress
2026-09-27 02:40:36.276 UTC [12381] WARNING: there is no transaction in progress
2026-09-27 02:40:36.291 UTC [12381] WARNING: there is no transaction in progress
2026-09-27 02:40:36.306 UTC [12381] WARNING: there is no transaction in progress
2026-09-27 02:40:36.322 UTC [12381] WARNING: there is no transaction in progress
2026-09-27 02:40:36.380 UTC [12381] WARNING: there is no transaction in progress
2026-09-27 02:40:36.386 UTC [12381] WARNING: there is no transaction in progress
2026-09-27 02:40:36.399 UTC [12381] WARNING: there is no transaction in progress
2026-09-27 02:40:36.408 UTC [12381] WARNING: there is no transaction in progress
2026-09-27 02:40:36.417 UTC [12381] WARNING: there is no transaction in progress
2026-09-27 02:40:36.430 UTC [12381] WARNING: there is no transaction in progress
2026-09-27 02:40:36.439 UTC [12381] WARNING: there is no transaction in progress
2026-09-27 02:40:36.445 UTC [12381] WARNING: there is no transaction in progress
2026-09-27 02:40:36.452 UTC [12381] WARNING: there is no transaction in progress
2026-09-27 02:40:36.458 UTC [12381] WARNING: there is no transaction in progress
2026-09-27 02:40:37,040 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 02:40:37,040 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBFiRWZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBFiRWZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 02:40:37,285 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 02:40:37,286 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBFiRWZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBFiRWZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 02:40:37,514 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 02:40:37,515 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBFiRWZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBFiRWZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 02:40:37,743 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 02:40:37,743 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBFiRWZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBFiRWZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 02:40:37,989 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBFiRWZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBFiRWZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 02:40:38,231 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBFiRWZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBFiRWZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-zv8z.c.gcp-upstream.internal';"
2026-09-27 02:40:38,449 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 02:40:38,450 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBFiRWZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBFiRWZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53069/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 02:40:38,700 [I] DB_PATCHES have already been applied
I20260927 02:40:38.705391 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskBFiRWZ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskBFiRWZ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskBFiRWZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBFiRWZ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 02:40:38.705454 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260927 02:40:38.708598 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:59177
Sep 27, 2026 2:40:38 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 2:40:38 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 2:40:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 2:40:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 2:40:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 2:40:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 2:40:39 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 02:40:39.706322 392 mini_ranger.cc:160] Time spent starting Ranger: real 7.970s user 0.000s sys 0.009s
Sep 27, 2026 2:40:58 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 2:40:58 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 27, 2026 2:40:58 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 2:40:58 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 2:40:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 2:40:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 2:40:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 2:40:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 2:40:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 2:40:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 2:40:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 2:40:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 2:40:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 2:40:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 2:40:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 2:40:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 2:40:58 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 2:40:59 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 02:41:00.006773 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 02:41:00 dist-test-slave-zv8z krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476860, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 02:41:00.098506 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 02:41:00.107537 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 02:41:01.092901 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 02:41:01.092984 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ranger-kms
I20260927 02:41:01.093001 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:53069
2026-09-27 02:41:01,241 [I] DB FLAVOR :POSTGRES
2026-09-27 02:41:01,241 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 02:41:01,241 [I] Checking connection
2026-09-27 02:41:01,242 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBFiRWZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53069/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 02:41:01,470 [I] connection success
2026-09-27 02:41:01,470 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 02:41:01,470 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 02:41:01,470 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBFiRWZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53069/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 02:41:01,692 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 02:41:01,693 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 02:41:01,693 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 02:41:01,693 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBFiRWZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53069/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 02:41:01,932 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 02:41:02.396014 392 mini_ranger_kms.cc:310] Created kms service
I20260927 02:41:02.597796 392 mini_ranger_kms.cc:326] Created kudu user
I20260927 02:41:02.637459 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 02:41:02 dist-test-slave-zv8z krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476860, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:41:03.006803 392 mini_ranger_kms.cc:384] Added ranger policy
I20260927 02:41:03.007040 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskBFiRWZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBFiRWZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskBFiRWZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 02:41:03.007066 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260927 02:41:03.010653 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12561
I20260927 02:41:03.010735 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:38527
Sep 27, 2026 2:41:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 2:41:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 2:41:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 2:41:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 2:41:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 2:41:04 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 02:41:04.976445 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.878s user 0.014s sys 0.119s
I20260927 02:41:04.976555 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 02:41:04.976629 392 mini_ranger_kms.cc:401] 127.0.98.20:38527/kms/v1/keys
Sep 27 02:41:05 dist-test-slave-zv8z krb5kdc[12168](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790476865, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 02:41:05 dist-test-slave-zv8z krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476865, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 02:41:05 dist-test-slave-zv8z krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476865, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 02:41:08 dist-test-slave-zv8z krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:41:08.098006 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42643
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42643
--ranger_config_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:38527/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:41:08.215229 12612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:41:08.215504 12612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:41:08.215572 12612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:41:08.219638 12612 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:41:08.219702 12612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:41:08.219728 12612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:41:08.219750 12612 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:41:08.219780 12612 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:41:08.225481 12612 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:38527/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42643
--ranger_config_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:41:08.226596 12612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:41:08.227607 12612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:41:08.236284 12620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:41:08.236428 12618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:41:08.236526 12612 server_base.cc:1034] running on GCE node
W20260927 02:41:08.236322 12617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:41:08.237322 12612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:41:08.238866 12612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:41:08.240079 12612 hybrid_clock.cc:648] HybridClock initialized: now 1790476868240052 us; error 42 us; skew 500 ppm
Sep 27 02:41:08 dist-test-slave-zv8z krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:41:08.246706 12612 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:41:08.247700 12612 webserver.cc:460] Webserver started at http://127.0.98.62:41699/ using document root <none> and password file <none>
I20260927 02:41:08.248238 12612 fs_manager.cc:359] Metadata directory not provided
I20260927 02:41:08.248301 12612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:41:08.248492 12612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 02:41:08 dist-test-slave-zv8z krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:41:08.310118 12612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/master-0/data/instance:
uuid: "acea6dc179644646b61d266863dde992"
format_stamp: "Formatted at 2026-09-27 02:41:08 on dist-test-slave-zv8z"
server_key: "14c160500d8990dae41de46b72f5a111"
server_key_iv: "c2d607dbd46b1a4a8d3c51ded092f359"
server_key_version: "kuduclusterkey@0"
I20260927 02:41:08.310765 12612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/master-0/wal/instance:
uuid: "acea6dc179644646b61d266863dde992"
format_stamp: "Formatted at 2026-09-27 02:41:08 on dist-test-slave-zv8z"
server_key: "14c160500d8990dae41de46b72f5a111"
server_key_iv: "c2d607dbd46b1a4a8d3c51ded092f359"
server_key_version: "kuduclusterkey@0"
I20260927 02:41:08.314898 12612 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 02:41:08.332588 12627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:08.333945 12612 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260927 02:41:08.334070 12612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "acea6dc179644646b61d266863dde992"
format_stamp: "Formatted at 2026-09-27 02:41:08 on dist-test-slave-zv8z"
server_key: "14c160500d8990dae41de46b72f5a111"
server_key_iv: "c2d607dbd46b1a4a8d3c51ded092f359"
server_key_version: "kuduclusterkey@0"
I20260927 02:41:08.334221 12612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:41:08.355177 12612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:41:08.358135 12612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:41:08.358345 12612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:41:08.365839 12612 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42643
I20260927 02:41:08.365864 12679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42643 every 8 connection(s)
I20260927 02:41:08.366859 12612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:41:08.369549 12680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:41:08.374950 12680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acea6dc179644646b61d266863dde992: Bootstrap starting.
I20260927 02:41:08.375833 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 12612
I20260927 02:41:08.375950 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/master-0/wal/instance
I20260927 02:41:08.376233 392 external_mini_cluster.cc:1428] Setting key 3eeb4a7a27a3baf0ce37ce4158df8b3b
I20260927 02:41:08.377604 12680 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P acea6dc179644646b61d266863dde992: Neither blocks nor log segments found. Creating new log.
I20260927 02:41:08.378552 12680 log.cc:824] T 00000000000000000000000000000000 P acea6dc179644646b61d266863dde992: Log is configured to *not* fsync() on all Append() calls
I20260927 02:41:08.380640 12680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acea6dc179644646b61d266863dde992: No bootstrap required, opened a new log
I20260927 02:41:08.384582 12680 raft_consensus.cc:348] T 00000000000000000000000000000000 P acea6dc179644646b61d266863dde992 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acea6dc179644646b61d266863dde992" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42643 } }
I20260927 02:41:08.384811 12680 raft_consensus.cc:374] T 00000000000000000000000000000000 P acea6dc179644646b61d266863dde992 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:41:08.384881 12680 raft_consensus.cc:729] T 00000000000000000000000000000000 P acea6dc179644646b61d266863dde992 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acea6dc179644646b61d266863dde992, State: Initialized, Role: FOLLOWER
I20260927 02:41:08.385306 12680 consensus_queue.cc:260] T 00000000000000000000000000000000 P acea6dc179644646b61d266863dde992 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acea6dc179644646b61d266863dde992" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42643 } }
I20260927 02:41:08.385432 12680 raft_consensus.cc:388] T 00000000000000000000000000000000 P acea6dc179644646b61d266863dde992 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:41:08.385507 12680 raft_consensus.cc:482] T 00000000000000000000000000000000 P acea6dc179644646b61d266863dde992 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:41:08.385604 12680 raft_consensus.cc:3037] T 00000000000000000000000000000000 P acea6dc179644646b61d266863dde992 [term 0 FOLLOWER]: Advancing to term 1
Sep 27 02:41:08 dist-test-slave-zv8z krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:41:08.386529 12680 raft_consensus.cc:504] T 00000000000000000000000000000000 P acea6dc179644646b61d266863dde992 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acea6dc179644646b61d266863dde992" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42643 } }
I20260927 02:41:08.386941 12680 leader_election.cc:302] T 00000000000000000000000000000000 P acea6dc179644646b61d266863dde992 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: acea6dc179644646b61d266863dde992; no voters:
I20260927 02:41:08.387250 12680 leader_election.cc:288] T 00000000000000000000000000000000 P acea6dc179644646b61d266863dde992 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:41:08.387378 12685 raft_consensus.cc:2781] T 00000000000000000000000000000000 P acea6dc179644646b61d266863dde992 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:41:08.387862 12685 raft_consensus.cc:686] T 00000000000000000000000000000000 P acea6dc179644646b61d266863dde992 [term 1 LEADER]: Becoming Leader. State: Replica: acea6dc179644646b61d266863dde992, State: Running, Role: LEADER
I20260927 02:41:08.388200 12685 consensus_queue.cc:237] T 00000000000000000000000000000000 P acea6dc179644646b61d266863dde992 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acea6dc179644646b61d266863dde992" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42643 } }
I20260927 02:41:08.388605 12680 sys_catalog.cc:564] T 00000000000000000000000000000000 P acea6dc179644646b61d266863dde992 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:41:08.389369 12685 sys_catalog.cc:455] T 00000000000000000000000000000000 P acea6dc179644646b61d266863dde992 [sys.catalog]: SysCatalogTable state changed. Reason: New leader acea6dc179644646b61d266863dde992. Latest consensus state: current_term: 1 leader_uuid: "acea6dc179644646b61d266863dde992" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acea6dc179644646b61d266863dde992" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42643 } } }
I20260927 02:41:08.389477 12685 sys_catalog.cc:458] T 00000000000000000000000000000000 P acea6dc179644646b61d266863dde992 [sys.catalog]: This master's current role is: LEADER
I20260927 02:41:08.390950 12680 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 02:41:08.392273 12691 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:41:08.394697 12686 sys_catalog.cc:455] T 00000000000000000000000000000000 P acea6dc179644646b61d266863dde992 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "acea6dc179644646b61d266863dde992" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acea6dc179644646b61d266863dde992" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42643 } } }
I20260927 02:41:08.394995 12686 sys_catalog.cc:458] T 00000000000000000000000000000000 P acea6dc179644646b61d266863dde992 [sys.catalog]: This master's current role is: LEADER
I20260927 02:41:08.395339 12683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:08.377504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48456 (local address 127.0.98.62:42643)
0927 02:41:08.377950 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:08.377967 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:08.378008 (+ 41us) server_negotiation.cc:407] Connection header received
0927 02:41:08.378844 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:08.378871 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:08.379230 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:08.379564 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:41:08.380350 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:08.381731 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:08.382899 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:08.383611 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:08.383768 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:08.386609 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:41:08.386616 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:41:08.386634 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:41:08.386664 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:41:08.388663 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:08.389233 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:08.389239 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:08.389245 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:08.389379 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:08.389687 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:08.389690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:08.389692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:08.390212 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:41:08.390427 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:08.391255 (+ 828us) server_negotiation.cc:299] Negotiation successful
0927 02:41:08.391522 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":142,"threads_started":1}
I20260927 02:41:08.398809 12691 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:41:08.405997 12691 catalog_manager.cc:1347] Generated new cluster ID: 70767b28ac884081b17d137698f078c3
I20260927 02:41:08.406055 12691 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:41:08.413017 12691 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:41:08.414389 12691 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:41:08.424717 12691 catalog_manager.cc:5868] T 00000000000000000000000000000000 P acea6dc179644646b61d266863dde992: Generated new TSK 0
I20260927 02:41:08.425693 12691 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:41:09.125403 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:38527/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42643
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:41:09.263821 12725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:41:09.264147 12725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:41:09.264194 12725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:41:09.269393 12725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:41:09.269462 12725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:41:09.269526 12725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:41:09.275352 12725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:38527/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:41:09.276490 12725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:41:09.277724 12725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:41:09.290129 12735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:41:09.291082 12732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:41:09.291169 12733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:41:09.291328 12725 server_base.cc:1034] running on GCE node
I20260927 02:41:09.291888 12725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:41:09.292649 12725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:41:09.293814 12725 hybrid_clock.cc:648] HybridClock initialized: now 1790476869293785 us; error 47 us; skew 500 ppm
Sep 27 02:41:09 dist-test-slave-zv8z krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476869, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:41:09.297214 12725 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:41:09.298358 12725 webserver.cc:460] Webserver started at http://127.0.98.1:44793/ using document root <none> and password file <none>
I20260927 02:41:09.299104 12725 fs_manager.cc:359] Metadata directory not provided
I20260927 02:41:09.299201 12725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:41:09.299469 12725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 02:41:09 dist-test-slave-zv8z krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476869, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:41:09.315093 12725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-0/data/instance:
uuid: "6046a148cd2749b2a3da0b64e879bc6a"
format_stamp: "Formatted at 2026-09-27 02:41:09 on dist-test-slave-zv8z"
server_key: "43afc38873bb0c0596a1cee55e624205"
server_key_iv: "adbc2e921498ef92b9763bd7c28a4cb5"
server_key_version: "kuduclusterkey@0"
I20260927 02:41:09.315932 12725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6046a148cd2749b2a3da0b64e879bc6a"
format_stamp: "Formatted at 2026-09-27 02:41:09 on dist-test-slave-zv8z"
server_key: "43afc38873bb0c0596a1cee55e624205"
server_key_iv: "adbc2e921498ef92b9763bd7c28a4cb5"
server_key_version: "kuduclusterkey@0"
I20260927 02:41:09.324081 12725 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.000s
I20260927 02:41:09.342499 12742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:09.344304 12725 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260927 02:41:09.344508 12725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-0/wal
uuid: "6046a148cd2749b2a3da0b64e879bc6a"
format_stamp: "Formatted at 2026-09-27 02:41:09 on dist-test-slave-zv8z"
server_key: "43afc38873bb0c0596a1cee55e624205"
server_key_iv: "adbc2e921498ef92b9763bd7c28a4cb5"
server_key_version: "kuduclusterkey@0"
I20260927 02:41:09.344725 12725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:41:09.363261 12725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:41:09.366881 12725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:41:09.367118 12725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:41:09.367815 12725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:41:09.368842 12725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:41:09.368937 12725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:09.369056 12725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:41:09.369135 12725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:09.378957 12725 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35399
I20260927 02:41:09.380069 12725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:41:09.383601 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 12725
I20260927 02:41:09.383721 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:41:09.384053 392 external_mini_cluster.cc:1428] Setting key 6985e9a25991262fbc8be4cf7448682f
I20260927 02:41:09.389514 12855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35399 every 8 connection(s)
Sep 27 02:41:09 dist-test-slave-zv8z krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476869, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:41:09.401607 12861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:09.383115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42643 (local address 127.0.98.1:38579)
0927 02:41:09.384882 (+ 1767us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:09.384940 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 02:41:09.386543 (+ 1603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:41:09.388982 (+ 2439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:41:09.388992 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:41:09.389332 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:41:09.390037 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:09.390056 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:09.391267 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:09.391270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:09.392672 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:09.392683 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:09.393366 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:09.393371 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:09.393569 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:09.394281 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:41:09.394302 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:41:09.396036 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:41:09.398327 (+ 2291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:09.398333 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:09.398351 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:09.398670 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:09.398999 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:09.399002 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:09.399004 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:09.399180 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:09.399654 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:41:09.399661 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:41:09.399996 (+ 335us) client_negotiation.cc:769] Sending connection context
0927 02:41:09.400188 (+ 192us) client_negotiation.cc:240] Negotiation successful
0927 02:41:09.400516 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1536,"thread_start_us":850,"threads_started":1}
I20260927 02:41:09.402366 12860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:09.382573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38579 (local address 127.0.98.62:42643)
0927 02:41:09.388549 (+ 5976us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:09.388557 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:09.388573 (+ 16us) server_negotiation.cc:407] Connection header received
0927 02:41:09.388612 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:09.388615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:09.388661 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:09.388760 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:41:09.390231 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:09.391118 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:09.392845 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:09.393192 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:09.396214 (+ 3022us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:09.396346 (+ 132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:41:09.396350 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:41:09.396352 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:41:09.396369 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:41:09.398177 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:09.398806 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:09.398809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:09.398811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:09.398897 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:09.399296 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:09.399299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:09.399301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:09.399521 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:41:09.401952 (+ 2431us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:09.402030 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 02:41:09.402160 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5896,"thread_start_us":101,"threads_started":1}
I20260927 02:41:09.403635 12856 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42643
I20260927 02:41:09.403900 12856 heartbeater.cc:464] Registering TS with master...
I20260927 02:41:09.404443 12856 heartbeater.cc:511] Master 127.0.98.62:42643 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 02:41:09.406011 12644 ts_manager.cc:194] Registered new tserver with Master: 6046a148cd2749b2a3da0b64e879bc6a (127.0.98.1:35399)
I20260927 02:41:09.407196 12644 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38579
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:41:09.425467 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:38527/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42643
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 02:41:09 dist-test-slave-zv8z krb5kdc[12168](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790476869, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 02:41:09.581700 12864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:41:09.582084 12864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:41:09.582190 12864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:41:09.588110 12864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:41:09.588194 12864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:41:09.588294 12864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:41:09.594889 12864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:38527/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:41:09.596113 12864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:41:09.597148 12864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:41:09.606516 12870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:41:09.606516 12873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:41:09.606542 12871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:41:09.607677 12864 server_base.cc:1034] running on GCE node
I20260927 02:41:09.608060 12864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:41:09.608650 12864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:41:09.611967 12864 hybrid_clock.cc:648] HybridClock initialized: now 1790476869611945 us; error 27 us; skew 500 ppm
Sep 27 02:41:09 dist-test-slave-zv8z krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476869, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:41:09.615427 12864 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:41:09.616261 12864 webserver.cc:460] Webserver started at http://127.0.98.2:36617/ using document root <none> and password file <none>
I20260927 02:41:09.616780 12864 fs_manager.cc:359] Metadata directory not provided
I20260927 02:41:09.616844 12864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:41:09.617024 12864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 02:41:09 dist-test-slave-zv8z krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476869, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:41:09.634498 12864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-1/data/instance:
uuid: "b3040a41717b441385e3660424d9228f"
format_stamp: "Formatted at 2026-09-27 02:41:09 on dist-test-slave-zv8z"
server_key: "1e1b9b708d47c417d29d1abb648a30f6"
server_key_iv: "c205eb4f791cc3a465a5a782b311411e"
server_key_version: "kuduclusterkey@0"
I20260927 02:41:09.635329 12864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b3040a41717b441385e3660424d9228f"
format_stamp: "Formatted at 2026-09-27 02:41:09 on dist-test-slave-zv8z"
server_key: "1e1b9b708d47c417d29d1abb648a30f6"
server_key_iv: "c205eb4f791cc3a465a5a782b311411e"
server_key_version: "kuduclusterkey@0"
I20260927 02:41:09.643733 12864 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.000s
I20260927 02:41:09.669332 12880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:09.671272 12864 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.001s sys 0.000s
I20260927 02:41:09.671471 12864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-1/wal
uuid: "b3040a41717b441385e3660424d9228f"
format_stamp: "Formatted at 2026-09-27 02:41:09 on dist-test-slave-zv8z"
server_key: "1e1b9b708d47c417d29d1abb648a30f6"
server_key_iv: "c205eb4f791cc3a465a5a782b311411e"
server_key_version: "kuduclusterkey@0"
I20260927 02:41:09.671710 12864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:41:09.689442 12864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:41:09.697692 12864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:41:09.697959 12864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:41:09.698750 12864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:41:09.699847 12864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:41:09.699950 12864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:09.700048 12864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:41:09.700114 12864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:09.710496 12864 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38381
I20260927 02:41:09.711684 12864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:41:09.713925 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 12864
I20260927 02:41:09.714044 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:41:09.714589 392 external_mini_cluster.cc:1428] Setting key 3431b15aa76dee3df8b730914ea01adc
I20260927 02:41:09.730458 12993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38381 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Sep 27 02:41:09 dist-test-slave-zv8z krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476869, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 02:41:09.747447 12860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:09.734087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58437 (local address 127.0.98.62:42643)
0927 02:41:09.734223 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:09.734228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:09.734727 (+ 499us) server_negotiation.cc:407] Connection header received
0927 02:41:09.736011 (+ 1284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:09.736015 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:09.736072 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:09.736128 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:41:09.737794 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:09.738637 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:09.740333 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:09.740690 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:09.740767 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:09.743749 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:41:09.743753 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:41:09.743756 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:41:09.743787 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:41:09.745402 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:09.745864 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:09.745868 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:09.745870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:09.745942 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:09.746283 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:09.746286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:09.746288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:09.746458 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:41:09.746541 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:09.747155 (+ 614us) server_negotiation.cc:299] Negotiation successful
0927 02:41:09.747285 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 02:41:09.748971 12997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:09.732956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42643 (local address 127.0.98.2:58437)
0927 02:41:09.734557 (+ 1601us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:09.734621 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 02:41:09.735748 (+ 1127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:41:09.736573 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:41:09.736583 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:41:09.736933 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:41:09.737620 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:09.737641 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:09.738798 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:09.738802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:09.740195 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:09.740207 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:09.740970 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:09.740974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:09.741183 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:09.741854 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:41:09.741876 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:41:09.743551 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:41:09.745508 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:09.745514 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:09.745531 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:09.745775 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:09.746022 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:09.746024 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:09.746026 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:09.746178 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:09.746554 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:41:09.746561 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:41:09.746890 (+ 329us) client_negotiation.cc:769] Sending connection context
0927 02:41:09.747659 (+ 769us) client_negotiation.cc:240] Negotiation successful
0927 02:41:09.747977 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1360,"thread_start_us":161,"threads_started":1}
I20260927 02:41:09.750141 12994 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42643
I20260927 02:41:09.750411 12994 heartbeater.cc:464] Registering TS with master...
I20260927 02:41:09.750907 12994 heartbeater.cc:511] Master 127.0.98.62:42643 requested a full tablet report, sending...
I20260927 02:41:09.751856 12644 ts_manager.cc:194] Registered new tserver with Master: b3040a41717b441385e3660424d9228f (127.0.98.2:38381)
I20260927 02:41:09.752394 12644 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58437
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:41:09.775077 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:38527/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42643
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:41:09.945051 12999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:41:09.945461 12999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:41:09.945580 12999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:41:09.951195 12999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:41:09.951364 12999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:41:09.951488 12999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:41:09.959059 12999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:38527/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:41:09.960505 12999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:41:09.961802 12999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:41:09.974656 13005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:41:09.975018 12999 server_base.cc:1034] running on GCE node
W20260927 02:41:09.975543 13004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:41:09.981403 13007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:41:09.982028 12999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:41:09.982904 12999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:41:09.987988 12999 hybrid_clock.cc:648] HybridClock initialized: now 1790476869987901 us; error 106 us; skew 500 ppm
Sep 27 02:41:09 dist-test-slave-zv8z krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476869, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:41:09.991991 12999 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:41:09.993140 12999 webserver.cc:460] Webserver started at http://127.0.98.3:37583/ using document root <none> and password file <none>
I20260927 02:41:09.993966 12999 fs_manager.cc:359] Metadata directory not provided
I20260927 02:41:09.994105 12999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:41:09.994408 12999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 02:41:09 dist-test-slave-zv8z krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476869, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:41:10.010593 12999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-2/data/instance:
uuid: "5852087793e042adad05275d4d232408"
format_stamp: "Formatted at 2026-09-27 02:41:10 on dist-test-slave-zv8z"
server_key: "15be74af384d3e60c43fd7e39665ef3f"
server_key_iv: "ba8aa34ddf3dcd1db1d1a3651d4669f0"
server_key_version: "kuduclusterkey@0"
I20260927 02:41:10.011443 12999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5852087793e042adad05275d4d232408"
format_stamp: "Formatted at 2026-09-27 02:41:10 on dist-test-slave-zv8z"
server_key: "15be74af384d3e60c43fd7e39665ef3f"
server_key_iv: "ba8aa34ddf3dcd1db1d1a3651d4669f0"
server_key_version: "kuduclusterkey@0"
I20260927 02:41:10.019773 12999 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.000s
I20260927 02:41:10.042236 13014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:10.043944 12999 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.004s sys 0.000s
I20260927 02:41:10.044067 12999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-2/wal
uuid: "5852087793e042adad05275d4d232408"
format_stamp: "Formatted at 2026-09-27 02:41:10 on dist-test-slave-zv8z"
server_key: "15be74af384d3e60c43fd7e39665ef3f"
server_key_iv: "ba8aa34ddf3dcd1db1d1a3651d4669f0"
server_key_version: "kuduclusterkey@0"
I20260927 02:41:10.044191 12999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 27 02:41:10 dist-test-slave-zv8z krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476869, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:41:10.081449 12999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:41:10.085732 12999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:41:10.085999 12999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:41:10.086794 12999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:41:10.087867 12999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:41:10.087970 12999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:10.088068 12999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:41:10.088128 12999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:10.101140 12999 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36477
I20260927 02:41:10.102412 12999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:41:10.105151 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 12999
I20260927 02:41:10.105290 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:41:10.105636 392 external_mini_cluster.cc:1428] Setting key 3f945e851267144aee15fdc9bc4fc515
I20260927 02:41:10.112849 13127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36477 every 8 connection(s)
Sep 27 02:41:10 dist-test-slave-zv8z krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476869, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:41:10.141135 12860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:10.114801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48819 (local address 127.0.98.62:42643)
0927 02:41:10.114932 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:10.114937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:10.116632 (+ 1695us) server_negotiation.cc:407] Connection header received
0927 02:41:10.117835 (+ 1203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:10.117839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:10.117897 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:10.117957 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:41:10.121169 (+ 3212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:10.121926 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:10.129651 (+ 7725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:10.129960 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:10.130044 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:10.136514 (+ 6470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:41:10.136519 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:41:10.136522 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:41:10.136554 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:41:10.138410 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:10.139234 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:10.139238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:10.139240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:10.139330 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:10.139752 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:10.139756 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:10.139758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:10.139986 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:41:10.140064 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:10.140822 (+ 758us) server_negotiation.cc:299] Negotiation successful
0927 02:41:10.140953 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 02:41:10.144691 13130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:10.115339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42643 (local address 127.0.98.3:48819)
0927 02:41:10.116331 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:10.116399 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0927 02:41:10.117406 (+ 1007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:41:10.119942 (+ 2536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:41:10.119953 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:41:10.120324 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:41:10.120994 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:10.121015 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:10.122090 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:10.122093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:10.129462 (+ 7369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:10.129477 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:10.130279 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:10.130284 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:10.130496 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:10.134445 (+ 3949us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:41:10.134469 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:41:10.136286 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:41:10.138613 (+ 2327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:10.138619 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:10.138681 (+ 62us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:10.139040 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:10.139434 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:10.139437 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:10.139439 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:10.139636 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:10.140141 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:41:10.140147 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:41:10.140511 (+ 364us) client_negotiation.cc:769] Sending connection context
0927 02:41:10.141584 (+ 1073us) client_negotiation.cc:240] Negotiation successful
0927 02:41:10.143586 (+ 2002us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":174,"threads_started":1}
I20260927 02:41:10.146435 13128 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42643
I20260927 02:41:10.146674 13128 heartbeater.cc:464] Registering TS with master...
I20260927 02:41:10.147240 13128 heartbeater.cc:511] Master 127.0.98.62:42643 requested a full tablet report, sending...
I20260927 02:41:10.148427 12644 ts_manager.cc:194] Registered new tserver with Master: 5852087793e042adad05275d4d232408 (127.0.98.3:36477)
I20260927 02:41:10.149189 12644 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48819
I20260927 02:41:10.154868 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:41:10.155560 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 12725
I20260927 02:41:10.162658 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 12864
I20260927 02:41:10.170265 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 12999
I20260927 02:41:10.177623 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 12612
I20260927 02:41:10.184842 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42643
--webserver_interface=127.0.98.62
--webserver_port=41699
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42643
--ranger_config_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:38527/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:38527/kms with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:41:10.314654 13131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:41:10.314898 13131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:41:10.314939 13131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:41:10.319023 13131 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:41:10.319074 13131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:41:10.319092 13131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:41:10.319108 13131 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:41:10.319137 13131 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:41:10.324730 13131 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:38527/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42643
--ranger_config_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:41:10.325739 13131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:41:10.326745 13131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:41:10.334794 13136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:41:10.334785 13137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:41:10.334756 13139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:41:10.334829 13131 server_base.cc:1034] running on GCE node
I20260927 02:41:10.335507 13131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:41:10.336516 13131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:41:10.337682 13131 hybrid_clock.cc:648] HybridClock initialized: now 1790476870337657 us; error 37 us; skew 500 ppm
Sep 27 02:41:10 dist-test-slave-zv8z krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476870, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:41:10.340260 13131 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:41:10.341044 13131 webserver.cc:460] Webserver started at http://127.0.98.62:41699/ using document root <none> and password file <none>
I20260927 02:41:10.341521 13131 fs_manager.cc:359] Metadata directory not provided
I20260927 02:41:10.341581 13131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 02:41:10 dist-test-slave-zv8z krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476870, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:41:10.401016 13131 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 02:41:10.403183 13147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:10.404428 13131 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:41:10.404531 13131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "acea6dc179644646b61d266863dde992"
format_stamp: "Formatted at 2026-09-27 02:41:08 on dist-test-slave-zv8z"
server_key: "14c160500d8990dae41de46b72f5a111"
server_key_iv: "c2d607dbd46b1a4a8d3c51ded092f359"
server_key_version: "kuduclusterkey@0"
I20260927 02:41:10.405040 13131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:41:10.420814 13131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:41:10.423508 13131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:41:10.423656 13131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:41:10.430420 13131 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42643
I20260927 02:41:10.430446 13199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42643 every 8 connection(s)
I20260927 02:41:10.431380 13131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:41:10.432869 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 13131
I20260927 02:41:10.433252 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:38527/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:35399
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42643
--webserver_port=44793
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:38527/kms with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:41:10.435575 13200 sys_catalog.cc:263] Verifying existing consensus state
I20260927 02:41:10.437160 13200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acea6dc179644646b61d266863dde992: Bootstrap starting.
I20260927 02:41:10.452560 13200 log.cc:824] T 00000000000000000000000000000000 P acea6dc179644646b61d266863dde992: Log is configured to *not* fsync() on all Append() calls
I20260927 02:41:10.458624 13200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acea6dc179644646b61d266863dde992: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:41:10.458949 13200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acea6dc179644646b61d266863dde992: Bootstrap complete.
I20260927 02:41:10.462760 13200 raft_consensus.cc:348] T 00000000000000000000000000000000 P acea6dc179644646b61d266863dde992 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acea6dc179644646b61d266863dde992" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42643 } }
I20260927 02:41:10.463131 13200 raft_consensus.cc:729] T 00000000000000000000000000000000 P acea6dc179644646b61d266863dde992 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: acea6dc179644646b61d266863dde992, State: Initialized, Role: FOLLOWER
I20260927 02:41:10.463562 13200 consensus_queue.cc:260] T 00000000000000000000000000000000 P acea6dc179644646b61d266863dde992 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acea6dc179644646b61d266863dde992" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42643 } }
I20260927 02:41:10.463682 13200 raft_consensus.cc:388] T 00000000000000000000000000000000 P acea6dc179644646b61d266863dde992 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:41:10.463745 13200 raft_consensus.cc:482] T 00000000000000000000000000000000 P acea6dc179644646b61d266863dde992 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:41:10.463833 13200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P acea6dc179644646b61d266863dde992 [term 1 FOLLOWER]: Advancing to term 2
I20260927 02:41:10.465561 13200 raft_consensus.cc:504] T 00000000000000000000000000000000 P acea6dc179644646b61d266863dde992 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acea6dc179644646b61d266863dde992" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42643 } }
I20260927 02:41:10.465858 13200 leader_election.cc:302] T 00000000000000000000000000000000 P acea6dc179644646b61d266863dde992 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: acea6dc179644646b61d266863dde992; no voters:
I20260927 02:41:10.466120 13200 leader_election.cc:288] T 00000000000000000000000000000000 P acea6dc179644646b61d266863dde992 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 02:41:10.466239 13204 raft_consensus.cc:2781] T 00000000000000000000000000000000 P acea6dc179644646b61d266863dde992 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 02:41:10.466490 13204 raft_consensus.cc:686] T 00000000000000000000000000000000 P acea6dc179644646b61d266863dde992 [term 2 LEADER]: Becoming Leader. State: Replica: acea6dc179644646b61d266863dde992, State: Running, Role: LEADER
I20260927 02:41:10.466869 13204 consensus_queue.cc:237] T 00000000000000000000000000000000 P acea6dc179644646b61d266863dde992 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acea6dc179644646b61d266863dde992" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42643 } }
I20260927 02:41:10.467214 13200 sys_catalog.cc:564] T 00000000000000000000000000000000 P acea6dc179644646b61d266863dde992 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:41:10.467866 13204 sys_catalog.cc:455] T 00000000000000000000000000000000 P acea6dc179644646b61d266863dde992 [sys.catalog]: SysCatalogTable state changed. Reason: New leader acea6dc179644646b61d266863dde992. Latest consensus state: current_term: 2 leader_uuid: "acea6dc179644646b61d266863dde992" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acea6dc179644646b61d266863dde992" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42643 } } }
I20260927 02:41:10.467968 13204 sys_catalog.cc:458] T 00000000000000000000000000000000 P acea6dc179644646b61d266863dde992 [sys.catalog]: This master's current role is: LEADER
I20260927 02:41:10.468312 13211 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:41:10.468869 13205 sys_catalog.cc:455] T 00000000000000000000000000000000 P acea6dc179644646b61d266863dde992 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "acea6dc179644646b61d266863dde992" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acea6dc179644646b61d266863dde992" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42643 } } }
I20260927 02:41:10.469036 13205 sys_catalog.cc:458] T 00000000000000000000000000000000 P acea6dc179644646b61d266863dde992 [sys.catalog]: This master's current role is: LEADER
I20260927 02:41:10.471408 13211 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:41:10.471951 13200 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 02:41:10.471992 13211 catalog_manager.cc:1259] Loaded cluster ID: 70767b28ac884081b17d137698f078c3
I20260927 02:41:10.472036 13211 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:41:10.474275 13211 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:41:10.477201 13211 catalog_manager.cc:5879] T 00000000000000000000000000000000 P acea6dc179644646b61d266863dde992: Loaded TSK: 0
I20260927 02:41:10.478958 13211 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 02:41:10.565657 13202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:41:10.566007 13202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:41:10.566084 13202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:41:10.572167 13202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:41:10.572247 13202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:41:10.572335 13202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:41:10.580049 13202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:38527/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:35399
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44793
--tserver_master_addrs=127.0.98.62:42643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:41:10.581460 13202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:41:10.582911 13202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:41:10.595467 13240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:41:10.595568 13237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:41:10.595880 13202 server_base.cc:1034] running on GCE node
W20260927 02:41:10.596139 13238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:41:10.596622 13202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:41:10.597363 13202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:41:10.598526 13202 hybrid_clock.cc:648] HybridClock initialized: now 1790476870598495 us; error 48 us; skew 500 ppm
Sep 27 02:41:10 dist-test-slave-zv8z krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476870, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:41:10.601786 13202 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:41:10.602865 13202 webserver.cc:460] Webserver started at http://127.0.98.1:44793/ using document root <none> and password file <none>
I20260927 02:41:10.603557 13202 fs_manager.cc:359] Metadata directory not provided
I20260927 02:41:10.603652 13202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 02:41:10 dist-test-slave-zv8z krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476870, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:41:10.634423 13202 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 02:41:10.637638 13248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:10.639285 13202 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 02:41:10.639424 13202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-0/wal
uuid: "6046a148cd2749b2a3da0b64e879bc6a"
format_stamp: "Formatted at 2026-09-27 02:41:09 on dist-test-slave-zv8z"
server_key: "43afc38873bb0c0596a1cee55e624205"
server_key_iv: "adbc2e921498ef92b9763bd7c28a4cb5"
server_key_version: "kuduclusterkey@0"
I20260927 02:41:10.640139 13202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:41:10.671072 13202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:41:10.674712 13202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:41:10.674911 13202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:41:10.675648 13202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:41:10.676690 13202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:41:10.676752 13202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:10.676805 13202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:41:10.676837 13202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:10.687093 13202 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35399
I20260927 02:41:10.687393 13361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35399 every 8 connection(s)
I20260927 02:41:10.688226 13202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:41:10.690909 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 13202
I20260927 02:41:10.691386 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:38527/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38381
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42643
--webserver_port=36617
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:38527/kms with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 02:41:10 dist-test-slave-zv8z krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476870, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:41:10.709205 13364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:10.690285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47333 (local address 127.0.98.62:42643)
0927 02:41:10.690905 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:10.690928 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:10.691360 (+ 432us) server_negotiation.cc:407] Connection header received
0927 02:41:10.692521 (+ 1161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:10.692533 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:10.692900 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:10.693246 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:41:10.694936 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:10.696460 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:10.698164 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:10.698943 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:10.699091 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:10.702000 (+ 2909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:41:10.702007 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:41:10.702024 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:41:10.702056 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:41:10.704025 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:10.704661 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:10.704668 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:10.704674 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:10.704816 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:10.705215 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:10.705218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:10.705220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:10.705717 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:41:10.705848 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:10.707972 (+ 2124us) server_negotiation.cc:299] Negotiation successful
0927 02:41:10.708264 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":114,"threads_started":1}
I20260927 02:41:10.710031 13365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:10.690434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42643 (local address 127.0.98.1:47333)
0927 02:41:10.691178 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:10.691229 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 02:41:10.692053 (+ 824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:41:10.693749 (+ 1696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:41:10.693759 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:41:10.694093 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:41:10.694744 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:10.694764 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:10.696616 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:10.696620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:10.698024 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:10.698036 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:10.699305 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:10.699309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:10.699531 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:10.700216 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:41:10.700238 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:41:10.701751 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:41:10.704166 (+ 2415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:10.704172 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:10.704189 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:10.704523 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:10.704922 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:10.704925 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:10.704927 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:10.705108 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:10.705904 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:41:10.705911 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:41:10.706240 (+ 329us) client_negotiation.cc:769] Sending connection context
0927 02:41:10.706496 (+ 256us) client_negotiation.cc:240] Negotiation successful
0927 02:41:10.706841 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":102,"threads_started":1}
I20260927 02:41:10.713088 13362 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42643
I20260927 02:41:10.713330 13362 heartbeater.cc:464] Registering TS with master...
I20260927 02:41:10.713908 13362 heartbeater.cc:511] Master 127.0.98.62:42643 requested a full tablet report, sending...
W20260927 02:41:10.716233 13362 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42643 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 02:41:10.716365 13362 heartbeater.cc:464] Registering TS with master...
I20260927 02:41:10.716552 13362 heartbeater.cc:511] Master 127.0.98.62:42643 requested a full tablet report, sending...
I20260927 02:41:10.717389 13362 heartbeater.cc:464] Registering TS with master...
I20260927 02:41:10.717555 13362 heartbeater.cc:511] Master 127.0.98.62:42643 requested a full tablet report, sending...
W20260927 02:41:10.718207 13362 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 02:41:10.850436 13366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:41:10.850776 13366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:41:10.850826 13366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:41:10.855983 13366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:41:10.856061 13366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:41:10.856138 13366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:41:10.862617 13366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:38527/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38381
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36617
--tserver_master_addrs=127.0.98.62:42643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:41:10.863678 13366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:41:10.864724 13366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:41:10.874401 13372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:41:10.874451 13374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:41:10.874740 13366 server_base.cc:1034] running on GCE node
W20260927 02:41:10.874814 13371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:41:10.875288 13366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:41:10.876051 13366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:41:10.877235 13366 hybrid_clock.cc:648] HybridClock initialized: now 1790476870877201 us; error 50 us; skew 500 ppm
Sep 27 02:41:10 dist-test-slave-zv8z krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476870, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:41:10.880764 13366 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:41:10.881891 13366 webserver.cc:460] Webserver started at http://127.0.98.2:36617/ using document root <none> and password file <none>
I20260927 02:41:10.882565 13366 fs_manager.cc:359] Metadata directory not provided
I20260927 02:41:10.882700 13366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 02:41:10 dist-test-slave-zv8z krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476870, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:41:10.915470 13366 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.003s sys 0.000s
I20260927 02:41:10.918509 13382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:10.920848 13366 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 02:41:10.920976 13366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-1/wal
uuid: "b3040a41717b441385e3660424d9228f"
format_stamp: "Formatted at 2026-09-27 02:41:09 on dist-test-slave-zv8z"
server_key: "1e1b9b708d47c417d29d1abb648a30f6"
server_key_iv: "c205eb4f791cc3a465a5a782b311411e"
server_key_version: "kuduclusterkey@0"
I20260927 02:41:10.921706 13366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:41:10.956131 13366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:41:10.959913 13366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:41:10.960114 13366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:41:10.960883 13366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:41:10.961990 13366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:41:10.962041 13366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:10.962093 13366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:41:10.962116 13366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:10.972460 13366 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38381
I20260927 02:41:10.973049 13496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38381 every 8 connection(s)
I20260927 02:41:10.973701 13366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:41:10.978997 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 13366
I20260927 02:41:10.979425 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:38527/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36477
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42643
--webserver_port=37583
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:38527/kms with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 02:41:10 dist-test-slave-zv8z krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476870, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:41:10.991256 13364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:10.976345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51475 (local address 127.0.98.62:42643)
0927 02:41:10.976501 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:10.976506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:10.977379 (+ 873us) server_negotiation.cc:407] Connection header received
0927 02:41:10.978584 (+ 1205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:10.978588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:10.978642 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:10.978695 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:41:10.980298 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:10.981120 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:10.982836 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:10.983194 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:10.983317 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:10.986189 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:41:10.986193 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:41:10.986197 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:41:10.986230 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:41:10.987942 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:10.988653 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:10.988656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:10.988658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:10.988753 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:10.989175 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:10.989178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:10.989180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:10.989415 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:41:10.989525 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:10.990956 (+ 1431us) server_negotiation.cc:299] Negotiation successful
0927 02:41:10.991087 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 02:41:10.992476 13499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:10.976750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42643 (local address 127.0.98.2:51475)
0927 02:41:10.977229 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:10.977280 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 02:41:10.978321 (+ 1041us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:41:10.979119 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:41:10.979128 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:41:10.979461 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:41:10.980126 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:10.980145 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:10.981274 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:10.981277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:10.982710 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:10.982720 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:10.983384 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:10.983388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:10.983573 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:10.984235 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:41:10.984255 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:41:10.985980 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:41:10.988087 (+ 2107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:10.988094 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:10.988111 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:10.988478 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:10.988866 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:10.988869 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:10.988872 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:10.989060 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:10.989579 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:41:10.989586 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:41:10.989928 (+ 342us) client_negotiation.cc:769] Sending connection context
0927 02:41:10.990155 (+ 227us) client_negotiation.cc:240] Negotiation successful
0927 02:41:10.990544 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":112,"threads_started":1}
I20260927 02:41:10.993794 13497 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42643
I20260927 02:41:10.994050 13497 heartbeater.cc:464] Registering TS with master...
I20260927 02:41:10.994719 13497 heartbeater.cc:511] Master 127.0.98.62:42643 requested a full tablet report, sending...
W20260927 02:41:10.996363 13497 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42643 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 02:41:10.996477 13497 heartbeater.cc:464] Registering TS with master...
I20260927 02:41:10.996622 13497 heartbeater.cc:511] Master 127.0.98.62:42643 requested a full tablet report, sending...
I20260927 02:41:10.997429 13497 heartbeater.cc:464] Registering TS with master...
I20260927 02:41:10.997588 13497 heartbeater.cc:511] Master 127.0.98.62:42643 requested a full tablet report, sending...
W20260927 02:41:10.998378 13497 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 02:41:11.104099 13500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:41:11.104346 13500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:41:11.104403 13500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:41:11.108531 13500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:41:11.108594 13500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:41:11.108672 13500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:41:11.115067 13500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:38527/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36477
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=37583
--tserver_master_addrs=127.0.98.62:42643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:41:11.116480 13500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:41:11.117859 13500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:41:11.131556 13505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:41:11.131567 13508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:41:11.131595 13506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:41:11.131906 13500 server_base.cc:1034] running on GCE node
I20260927 02:41:11.132741 13500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:41:11.133432 13500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:41:11.134599 13500 hybrid_clock.cc:648] HybridClock initialized: now 1790476871134554 us; error 57 us; skew 500 ppm
Sep 27 02:41:11 dist-test-slave-zv8z krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476871, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:41:11.137427 13500 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:41:11.138285 13500 webserver.cc:460] Webserver started at http://127.0.98.3:37583/ using document root <none> and password file <none>
I20260927 02:41:11.138819 13500 fs_manager.cc:359] Metadata directory not provided
I20260927 02:41:11.138890 13500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 02:41:11 dist-test-slave-zv8z krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476871, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:41:11.159929 13500 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.003s sys 0.000s
I20260927 02:41:11.162976 13516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:11.164268 13500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 02:41:11.164392 13500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-2/wal
uuid: "5852087793e042adad05275d4d232408"
format_stamp: "Formatted at 2026-09-27 02:41:10 on dist-test-slave-zv8z"
server_key: "15be74af384d3e60c43fd7e39665ef3f"
server_key_iv: "ba8aa34ddf3dcd1db1d1a3651d4669f0"
server_key_version: "kuduclusterkey@0"
I20260927 02:41:11.165123 13500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:41:11.188936 13500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:41:11.192950 13500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:41:11.193200 13500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:41:11.193958 13500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:41:11.194977 13500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:41:11.195111 13500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:11.195219 13500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:41:11.195257 13500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:11.205397 13500 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36477
I20260927 02:41:11.205489 13629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36477 every 8 connection(s)
I20260927 02:41:11.207187 13500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:41:11.210958 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 13500
Sep 27 02:41:11 dist-test-slave-zv8z krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476871, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:41:11.221308 13364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:11.209032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34201 (local address 127.0.98.62:42643)
0927 02:41:11.209156 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:11.209160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:11.210323 (+ 1163us) server_negotiation.cc:407] Connection header received
0927 02:41:11.211282 (+ 959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:11.211285 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:11.211341 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:11.211423 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:41:11.212648 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:11.213278 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:11.214754 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:11.214962 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:11.215043 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:11.217299 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:41:11.217303 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:41:11.217306 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:41:11.217331 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:41:11.219113 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:11.219643 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:11.219647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:11.219649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:11.219739 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:11.220154 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:11.220156 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:11.220157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:11.220324 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:41:11.220412 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:11.220994 (+ 582us) server_negotiation.cc:299] Negotiation successful
0927 02:41:11.221146 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 02:41:11.222393 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:11.209653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42643 (local address 127.0.98.3:34201)
0927 02:41:11.210167 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:11.210208 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:41:11.211066 (+ 858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:41:11.211626 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:41:11.211636 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:41:11.211968 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:41:11.212511 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:11.212525 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:11.213390 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:11.213394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:11.214644 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:11.214653 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:11.215097 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:11.215101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:11.215280 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:11.215821 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:41:11.215838 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:41:11.217126 (+ 1288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:41:11.219241 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:11.219248 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:11.219263 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:11.219551 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:11.219841 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:11.219844 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:11.219846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:11.220045 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:11.220431 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:41:11.220438 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:41:11.220760 (+ 322us) client_negotiation.cc:769] Sending connection context
0927 02:41:11.220982 (+ 222us) client_negotiation.cc:240] Negotiation successful
0927 02:41:11.221294 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":144,"threads_started":1}
I20260927 02:41:11.223973 13630 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42643
I20260927 02:41:11.224234 13630 heartbeater.cc:464] Registering TS with master...
I20260927 02:41:11.224815 13630 heartbeater.cc:511] Master 127.0.98.62:42643 requested a full tablet report, sending...
I20260927 02:41:11.226706 13634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:11.211655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55786 (local address 127.0.98.62:42643)
0927 02:41:11.212733 (+ 1078us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:11.212737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:11.212747 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:41:11.212774 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:11.212777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:11.212816 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:11.212900 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:41:11.213819 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:11.214625 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:11.217284 (+ 2659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:11.217603 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:11.217697 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:11.219424 (+ 1727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:41:11.219428 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:41:11.219431 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:41:11.219457 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:41:11.221381 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:11.222135 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:11.222139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:11.222141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:11.222232 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:11.225682 (+ 3450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:11.225685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:11.225687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:11.225913 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:41:11.226180 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:11.226419 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 02:41:11.226547 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1014,"thread_start_us":87,"threads_started":1}
W20260927 02:41:11.226958 13630 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42643 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 02:41:11.227073 13630 heartbeater.cc:464] Registering TS with master...
I20260927 02:41:11.227243 13630 heartbeater.cc:511] Master 127.0.98.62:42643 requested a full tablet report, sending...
I20260927 02:41:11.230075 13630 heartbeater.cc:464] Registering TS with master...
I20260927 02:41:11.230480 13630 heartbeater.cc:511] Master 127.0.98.62:42643 requested a full tablet report, sending...
W20260927 02:41:11.231245 13630 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 02:41:11 dist-test-slave-zv8z krb5kdc[12168](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790476871, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:41:11.718446 13362 heartbeater.cc:464] Registering TS with master...
I20260927 02:41:11.718760 13362 heartbeater.cc:511] Master 127.0.98.62:42643 requested a full tablet report, sending...
I20260927 02:41:11.720069 13164 ts_manager.cc:194] Registered new tserver with Master: 6046a148cd2749b2a3da0b64e879bc6a (127.0.98.1:35399)
I20260927 02:41:11.721738 13164 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47333
Sep 27 02:41:11 dist-test-slave-zv8z krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476871, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:41:11.998596 13497 heartbeater.cc:464] Registering TS with master...
I20260927 02:41:11.998890 13497 heartbeater.cc:511] Master 127.0.98.62:42643 requested a full tablet report, sending...
I20260927 02:41:11.999553 13164 ts_manager.cc:194] Registered new tserver with Master: b3040a41717b441385e3660424d9228f (127.0.98.2:38381)
I20260927 02:41:12.000309 13164 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51475
Sep 27 02:41:12 dist-test-slave-zv8z krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476871, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:41:12.231462 13630 heartbeater.cc:464] Registering TS with master...
I20260927 02:41:12.231767 13630 heartbeater.cc:511] Master 127.0.98.62:42643 requested a full tablet report, sending...
I20260927 02:41:12.232537 13164 ts_manager.cc:194] Registered new tserver with Master: 5852087793e042adad05275d4d232408 (127.0.98.3:36477)
I20260927 02:41:12.233335 13164 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34201
I20260927 02:41:12.236081 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:41:12.256228 13653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:12.237997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55788 (local address 127.0.98.62:42643)
0927 02:41:12.238586 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:12.238592 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:12.246581 (+ 7989us) server_negotiation.cc:407] Connection header received
0927 02:41:12.246727 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:12.246730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:12.246786 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:12.246872 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:41:12.247823 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:12.248675 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:12.249918 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:12.250343 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:12.250426 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:12.252225 (+ 1799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:41:12.252229 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:41:12.252232 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:41:12.252260 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:41:12.253970 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:12.254465 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:12.254469 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:12.254470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:12.254556 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:12.254828 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:12.254831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:12.254833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:12.255064 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:41:12.255185 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:12.255910 (+ 725us) server_negotiation.cc:299] Negotiation successful
0927 02:41:12.256035 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":104,"threads_started":1}
I20260927 02:41:12.260584 13164 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55788:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:41:12.265285 13164 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:41:12.290127 13660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:12.279004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36477 (local address 127.0.0.1:45434)
0927 02:41:12.282789 (+ 3785us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:12.282803 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:41:12.282913 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:41:12.283798 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:41:12.283804 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:41:12.283827 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:41:12.283962 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:12.283968 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:12.286141 (+ 2173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:12.286144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:12.287498 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:12.287504 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:12.289589 (+ 2085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:12.289593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:12.289704 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:12.289741 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 02:41:12.289844 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 02:41:12.289957 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3726,"mutex_wait_us":4,"thread_start_us":83,"threads_started":1}
I20260927 02:41:12.291013 13663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:12.279351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45434 (local address 127.0.98.3:36477)
0927 02:41:12.282658 (+ 3307us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:12.282666 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:12.283051 (+ 385us) server_negotiation.cc:407] Connection header received
0927 02:41:12.283129 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:12.283135 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:12.283352 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:12.283491 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:41:12.284093 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:12.286001 (+ 1908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:12.288417 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:12.289281 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:12.289382 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:12.289532 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:12.290769 (+ 1237us) server_negotiation.cc:299] Negotiation successful
0927 02:41:12.290878 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3195,"thread_start_us":68,"threads_started":1}
I20260927 02:41:12.294620 13564 tablet_service.cc:1467] Processing CreateTablet for tablet b49dc530f36b434ea58fe042ac0f0e23 (DEFAULT_TABLE table=test-table [id=2d2a2d9b5cc843b2937555acd9579c9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:41:12.295785 13564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b49dc530f36b434ea58fe042ac0f0e23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:41:12.298806 13664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:12.280587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35399 (local address 127.0.0.1:44040)
0927 02:41:12.286384 (+ 5797us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:12.286397 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:41:12.286478 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:41:12.287323 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:41:12.287326 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:41:12.287342 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:41:12.287426 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:12.287431 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:12.294359 (+ 6928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:12.294372 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:12.295619 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:12.295631 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:12.298338 (+ 2707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:12.298343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:12.298469 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:12.298501 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 02:41:12.298557 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 02:41:12.298648 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5735,"thread_start_us":78,"threads_started":1}
I20260927 02:41:12.299571 13665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:12.280949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44040 (local address 127.0.98.1:35399)
0927 02:41:12.286498 (+ 5549us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:12.286505 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:12.286535 (+ 30us) server_negotiation.cc:407] Connection header received
0927 02:41:12.286663 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:12.286669 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:12.286886 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:12.287044 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:41:12.290277 (+ 3233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:12.291444 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:12.295791 (+ 4347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:12.296644 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:12.299096 (+ 2452us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:12.299274 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:12.299326 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 02:41:12.299431 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5423,"thread_start_us":104,"threads_started":1}
I20260927 02:41:12.303077 13296 tablet_service.cc:1467] Processing CreateTablet for tablet b49dc530f36b434ea58fe042ac0f0e23 (DEFAULT_TABLE table=test-table [id=2d2a2d9b5cc843b2937555acd9579c9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:41:12.304052 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:12.278104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38381 (local address 127.0.0.1:44024)
0927 02:41:12.281612 (+ 3508us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:12.281665 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 02:41:12.283642 (+ 1977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:41:12.287529 (+ 3887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:41:12.287544 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:41:12.287559 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:41:12.287635 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:12.287641 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:12.288976 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:12.288979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:12.290023 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:12.290031 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:12.303598 (+ 13567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:12.303603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:12.303709 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:12.303737 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 02:41:12.303819 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 02:41:12.303907 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3304,"thread_start_us":90,"threads_started":1}
I20260927 02:41:12.304775 13661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:12.278208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44024 (local address 127.0.98.2:38381)
0927 02:41:12.286726 (+ 8518us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:12.286733 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:12.286758 (+ 25us) server_negotiation.cc:407] Connection header received
0927 02:41:12.286831 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:12.286836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:12.287028 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:12.287168 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:41:12.287753 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:12.288849 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:12.302350 (+ 13501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:12.303224 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:12.303358 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:12.303500 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:12.304515 (+ 1015us) server_negotiation.cc:299] Negotiation successful
0927 02:41:12.304640 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8408,"thread_start_us":89,"threads_started":1}
I20260927 02:41:12.305398 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b49dc530f36b434ea58fe042ac0f0e23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:41:12.307883 13431 tablet_service.cc:1467] Processing CreateTablet for tablet b49dc530f36b434ea58fe042ac0f0e23 (DEFAULT_TABLE table=test-table [id=2d2a2d9b5cc843b2937555acd9579c9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:41:12.308944 13431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b49dc530f36b434ea58fe042ac0f0e23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:41:12.318621 13666 tablet_bootstrap.cc:492] T b49dc530f36b434ea58fe042ac0f0e23 P b3040a41717b441385e3660424d9228f: Bootstrap starting.
I20260927 02:41:12.320971 13666 tablet_bootstrap.cc:654] T b49dc530f36b434ea58fe042ac0f0e23 P b3040a41717b441385e3660424d9228f: Neither blocks nor log segments found. Creating new log.
I20260927 02:41:12.321938 13666 log.cc:824] T b49dc530f36b434ea58fe042ac0f0e23 P b3040a41717b441385e3660424d9228f: Log is configured to *not* fsync() on all Append() calls
I20260927 02:41:12.322060 13668 tablet_bootstrap.cc:492] T b49dc530f36b434ea58fe042ac0f0e23 P 5852087793e042adad05275d4d232408: Bootstrap starting.
I20260927 02:41:12.324295 13668 tablet_bootstrap.cc:654] T b49dc530f36b434ea58fe042ac0f0e23 P 5852087793e042adad05275d4d232408: Neither blocks nor log segments found. Creating new log.
I20260927 02:41:12.324743 13667 tablet_bootstrap.cc:492] T b49dc530f36b434ea58fe042ac0f0e23 P 6046a148cd2749b2a3da0b64e879bc6a: Bootstrap starting.
I20260927 02:41:12.328655 13668 log.cc:824] T b49dc530f36b434ea58fe042ac0f0e23 P 5852087793e042adad05275d4d232408: Log is configured to *not* fsync() on all Append() calls
I20260927 02:41:12.329370 13666 tablet_bootstrap.cc:492] T b49dc530f36b434ea58fe042ac0f0e23 P b3040a41717b441385e3660424d9228f: No bootstrap required, opened a new log
I20260927 02:41:12.329396 13667 tablet_bootstrap.cc:654] T b49dc530f36b434ea58fe042ac0f0e23 P 6046a148cd2749b2a3da0b64e879bc6a: Neither blocks nor log segments found. Creating new log.
I20260927 02:41:12.329671 13666 ts_tablet_manager.cc:1397] T b49dc530f36b434ea58fe042ac0f0e23 P b3040a41717b441385e3660424d9228f: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260927 02:41:12.330214 13667 log.cc:824] T b49dc530f36b434ea58fe042ac0f0e23 P 6046a148cd2749b2a3da0b64e879bc6a: Log is configured to *not* fsync() on all Append() calls
I20260927 02:41:12.332741 13668 tablet_bootstrap.cc:492] T b49dc530f36b434ea58fe042ac0f0e23 P 5852087793e042adad05275d4d232408: No bootstrap required, opened a new log
I20260927 02:41:12.333045 13668 ts_tablet_manager.cc:1397] T b49dc530f36b434ea58fe042ac0f0e23 P 5852087793e042adad05275d4d232408: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260927 02:41:12.333956 13666 raft_consensus.cc:348] T b49dc530f36b434ea58fe042ac0f0e23 P b3040a41717b441385e3660424d9228f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852087793e042adad05275d4d232408" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36477 } } peers { permanent_uuid: "b3040a41717b441385e3660424d9228f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38381 } } peers { permanent_uuid: "6046a148cd2749b2a3da0b64e879bc6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35399 } }
I20260927 02:41:12.334374 13666 raft_consensus.cc:374] T b49dc530f36b434ea58fe042ac0f0e23 P b3040a41717b441385e3660424d9228f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:41:12.334466 13666 raft_consensus.cc:729] T b49dc530f36b434ea58fe042ac0f0e23 P b3040a41717b441385e3660424d9228f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3040a41717b441385e3660424d9228f, State: Initialized, Role: FOLLOWER
I20260927 02:41:12.335075 13666 consensus_queue.cc:260] T b49dc530f36b434ea58fe042ac0f0e23 P b3040a41717b441385e3660424d9228f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852087793e042adad05275d4d232408" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36477 } } peers { permanent_uuid: "b3040a41717b441385e3660424d9228f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38381 } } peers { permanent_uuid: "6046a148cd2749b2a3da0b64e879bc6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35399 } }
I20260927 02:41:12.336251 13666 ts_tablet_manager.cc:1428] T b49dc530f36b434ea58fe042ac0f0e23 P b3040a41717b441385e3660424d9228f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 02:41:12.338522 13668 raft_consensus.cc:348] T b49dc530f36b434ea58fe042ac0f0e23 P 5852087793e042adad05275d4d232408 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852087793e042adad05275d4d232408" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36477 } } peers { permanent_uuid: "b3040a41717b441385e3660424d9228f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38381 } } peers { permanent_uuid: "6046a148cd2749b2a3da0b64e879bc6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35399 } }
I20260927 02:41:12.338850 13668 raft_consensus.cc:374] T b49dc530f36b434ea58fe042ac0f0e23 P 5852087793e042adad05275d4d232408 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:41:12.338932 13668 raft_consensus.cc:729] T b49dc530f36b434ea58fe042ac0f0e23 P 5852087793e042adad05275d4d232408 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5852087793e042adad05275d4d232408, State: Initialized, Role: FOLLOWER
I20260927 02:41:12.339459 13668 consensus_queue.cc:260] T b49dc530f36b434ea58fe042ac0f0e23 P 5852087793e042adad05275d4d232408 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852087793e042adad05275d4d232408" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36477 } } peers { permanent_uuid: "b3040a41717b441385e3660424d9228f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38381 } } peers { permanent_uuid: "6046a148cd2749b2a3da0b64e879bc6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35399 } }
I20260927 02:41:12.341568 13668 ts_tablet_manager.cc:1428] T b49dc530f36b434ea58fe042ac0f0e23 P 5852087793e042adad05275d4d232408: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260927 02:41:12.343196 13667 tablet_bootstrap.cc:492] T b49dc530f36b434ea58fe042ac0f0e23 P 6046a148cd2749b2a3da0b64e879bc6a: No bootstrap required, opened a new log
I20260927 02:41:12.343459 13667 ts_tablet_manager.cc:1397] T b49dc530f36b434ea58fe042ac0f0e23 P 6046a148cd2749b2a3da0b64e879bc6a: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.000s
I20260927 02:41:12.346688 13497 heartbeater.cc:503] Master 127.0.98.62:42643 was elected leader, sending a full tablet report...
I20260927 02:41:12.349133 13667 raft_consensus.cc:348] T b49dc530f36b434ea58fe042ac0f0e23 P 6046a148cd2749b2a3da0b64e879bc6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852087793e042adad05275d4d232408" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36477 } } peers { permanent_uuid: "b3040a41717b441385e3660424d9228f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38381 } } peers { permanent_uuid: "6046a148cd2749b2a3da0b64e879bc6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35399 } }
I20260927 02:41:12.349457 13667 raft_consensus.cc:374] T b49dc530f36b434ea58fe042ac0f0e23 P 6046a148cd2749b2a3da0b64e879bc6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:41:12.349507 13667 raft_consensus.cc:729] T b49dc530f36b434ea58fe042ac0f0e23 P 6046a148cd2749b2a3da0b64e879bc6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6046a148cd2749b2a3da0b64e879bc6a, State: Initialized, Role: FOLLOWER
I20260927 02:41:12.350077 13667 consensus_queue.cc:260] T b49dc530f36b434ea58fe042ac0f0e23 P 6046a148cd2749b2a3da0b64e879bc6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852087793e042adad05275d4d232408" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36477 } } peers { permanent_uuid: "b3040a41717b441385e3660424d9228f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38381 } } peers { permanent_uuid: "6046a148cd2749b2a3da0b64e879bc6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35399 } }
I20260927 02:41:12.350690 13630 heartbeater.cc:503] Master 127.0.98.62:42643 was elected leader, sending a full tablet report...
I20260927 02:41:12.354148 13667 ts_tablet_manager.cc:1428] T b49dc530f36b434ea58fe042ac0f0e23 P 6046a148cd2749b2a3da0b64e879bc6a: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260927 02:41:12.354589 13362 heartbeater.cc:503] Master 127.0.98.62:42643 was elected leader, sending a full tablet report...
W20260927 02:41:12.440970 13363 tablet.cc:2369] T b49dc530f36b434ea58fe042ac0f0e23 P 6046a148cd2749b2a3da0b64e879bc6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:41:12.461202 13631 tablet.cc:2369] T b49dc530f36b434ea58fe042ac0f0e23 P 5852087793e042adad05275d4d232408: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:41:12.475926 13498 tablet.cc:2369] T b49dc530f36b434ea58fe042ac0f0e23 P b3040a41717b441385e3660424d9228f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:41:12.652091 13672 raft_consensus.cc:482] T b49dc530f36b434ea58fe042ac0f0e23 P b3040a41717b441385e3660424d9228f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:41:12.652297 13672 raft_consensus.cc:504] T b49dc530f36b434ea58fe042ac0f0e23 P b3040a41717b441385e3660424d9228f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852087793e042adad05275d4d232408" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36477 } } peers { permanent_uuid: "b3040a41717b441385e3660424d9228f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38381 } } peers { permanent_uuid: "6046a148cd2749b2a3da0b64e879bc6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35399 } }
I20260927 02:41:12.653487 13672 leader_election.cc:288] T b49dc530f36b434ea58fe042ac0f0e23 P b3040a41717b441385e3660424d9228f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5852087793e042adad05275d4d232408 (127.0.98.3:36477), 6046a148cd2749b2a3da0b64e879bc6a (127.0.98.1:35399)
I20260927 02:41:12.657807 13665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:12.654206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33487 (local address 127.0.98.1:35399)
0927 02:41:12.654345 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:12.654349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:12.654510 (+ 161us) server_negotiation.cc:407] Connection header received
0927 02:41:12.654664 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:12.654667 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:12.654715 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:12.654787 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:41:12.655092 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:12.655810 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:12.656840 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:12.657254 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:12.657353 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:12.657415 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:12.657613 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 02:41:12.657681 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 02:41:12.657831 13683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:12.653898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35399 (local address 127.0.98.2:33487)
0927 02:41:12.654407 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:12.654421 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:41:12.654547 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:41:12.654822 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:41:12.654826 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:41:12.654845 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:41:12.654973 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:12.654980 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:12.655909 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:12.655912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:12.656728 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:12.656736 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:12.657367 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:12.657371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:12.657479 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:12.657506 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 02:41:12.657581 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 02:41:12.657654 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"mutex_wait_us":2,"thread_start_us":106,"threads_started":1}
I20260927 02:41:12.658403 13682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:12.653684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36477 (local address 127.0.98.2:44929)
0927 02:41:12.654057 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:12.654073 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 02:41:12.654217 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:41:12.654649 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:41:12.654652 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:41:12.654668 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:41:12.654805 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:12.654811 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:12.655820 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:12.655823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:12.657159 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:12.657170 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:12.657998 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:12.658002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:12.658094 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:12.658115 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 02:41:12.658188 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 02:41:12.658272 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":45,"thread_start_us":115,"threads_started":1}
I20260927 02:41:12.658444 13663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:12.653995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44929 (local address 127.0.98.3:36477)
0927 02:41:12.654108 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:12.654112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:12.654180 (+ 68us) server_negotiation.cc:407] Connection header received
0927 02:41:12.654388 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:12.654392 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:12.654457 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:12.654505 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:41:12.654946 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:12.655693 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:12.657291 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:12.657792 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:12.657885 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:12.657939 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:12.658232 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 02:41:12.658331 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 02:41:12.658576 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b49dc530f36b434ea58fe042ac0f0e23" candidate_uuid: "b3040a41717b441385e3660424d9228f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6046a148cd2749b2a3da0b64e879bc6a" is_pre_election: true
I20260927 02:41:12.658921 13316 raft_consensus.cc:2454] T b49dc530f36b434ea58fe042ac0f0e23 P 6046a148cd2749b2a3da0b64e879bc6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3040a41717b441385e3660424d9228f in term 0.
I20260927 02:41:12.659158 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b49dc530f36b434ea58fe042ac0f0e23" candidate_uuid: "b3040a41717b441385e3660424d9228f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5852087793e042adad05275d4d232408" is_pre_election: true
I20260927 02:41:12.659403 13384 leader_election.cc:302] T b49dc530f36b434ea58fe042ac0f0e23 P b3040a41717b441385e3660424d9228f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6046a148cd2749b2a3da0b64e879bc6a, b3040a41717b441385e3660424d9228f; no voters:
I20260927 02:41:12.659456 13584 raft_consensus.cc:2454] T b49dc530f36b434ea58fe042ac0f0e23 P 5852087793e042adad05275d4d232408 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3040a41717b441385e3660424d9228f in term 0.
I20260927 02:41:12.659686 13672 raft_consensus.cc:2781] T b49dc530f36b434ea58fe042ac0f0e23 P b3040a41717b441385e3660424d9228f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:41:12.659771 13672 raft_consensus.cc:482] T b49dc530f36b434ea58fe042ac0f0e23 P b3040a41717b441385e3660424d9228f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:41:12.659823 13672 raft_consensus.cc:3037] T b49dc530f36b434ea58fe042ac0f0e23 P b3040a41717b441385e3660424d9228f [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:41:12.660852 13672 raft_consensus.cc:504] T b49dc530f36b434ea58fe042ac0f0e23 P b3040a41717b441385e3660424d9228f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852087793e042adad05275d4d232408" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36477 } } peers { permanent_uuid: "b3040a41717b441385e3660424d9228f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38381 } } peers { permanent_uuid: "6046a148cd2749b2a3da0b64e879bc6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35399 } }
I20260927 02:41:12.661178 13672 leader_election.cc:288] T b49dc530f36b434ea58fe042ac0f0e23 P b3040a41717b441385e3660424d9228f [CANDIDATE]: Term 1 election: Requested vote from peers 5852087793e042adad05275d4d232408 (127.0.98.3:36477), 6046a148cd2749b2a3da0b64e879bc6a (127.0.98.1:35399)
I20260927 02:41:12.661461 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b49dc530f36b434ea58fe042ac0f0e23" candidate_uuid: "b3040a41717b441385e3660424d9228f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5852087793e042adad05275d4d232408"
I20260927 02:41:12.661579 13584 raft_consensus.cc:3037] T b49dc530f36b434ea58fe042ac0f0e23 P 5852087793e042adad05275d4d232408 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:41:12.661573 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b49dc530f36b434ea58fe042ac0f0e23" candidate_uuid: "b3040a41717b441385e3660424d9228f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6046a148cd2749b2a3da0b64e879bc6a"
I20260927 02:41:12.661703 13316 raft_consensus.cc:3037] T b49dc530f36b434ea58fe042ac0f0e23 P 6046a148cd2749b2a3da0b64e879bc6a [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:41:12.662832 13316 raft_consensus.cc:2454] T b49dc530f36b434ea58fe042ac0f0e23 P 6046a148cd2749b2a3da0b64e879bc6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3040a41717b441385e3660424d9228f in term 1.
I20260927 02:41:12.662832 13584 raft_consensus.cc:2454] T b49dc530f36b434ea58fe042ac0f0e23 P 5852087793e042adad05275d4d232408 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3040a41717b441385e3660424d9228f in term 1.
I20260927 02:41:12.663133 13386 leader_election.cc:302] T b49dc530f36b434ea58fe042ac0f0e23 P b3040a41717b441385e3660424d9228f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5852087793e042adad05275d4d232408, b3040a41717b441385e3660424d9228f; no voters:
I20260927 02:41:12.663322 13672 raft_consensus.cc:2781] T b49dc530f36b434ea58fe042ac0f0e23 P b3040a41717b441385e3660424d9228f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:41:12.663537 13672 raft_consensus.cc:686] T b49dc530f36b434ea58fe042ac0f0e23 P b3040a41717b441385e3660424d9228f [term 1 LEADER]: Becoming Leader. State: Replica: b3040a41717b441385e3660424d9228f, State: Running, Role: LEADER
I20260927 02:41:12.663877 13672 consensus_queue.cc:237] T b49dc530f36b434ea58fe042ac0f0e23 P b3040a41717b441385e3660424d9228f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852087793e042adad05275d4d232408" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36477 } } peers { permanent_uuid: "b3040a41717b441385e3660424d9228f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38381 } } peers { permanent_uuid: "6046a148cd2749b2a3da0b64e879bc6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35399 } }
I20260927 02:41:12.667145 13163 catalog_manager.cc:5495] T b49dc530f36b434ea58fe042ac0f0e23 P b3040a41717b441385e3660424d9228f reported cstate change: term changed from 0 to 1, leader changed from <none> to b3040a41717b441385e3660424d9228f (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b3040a41717b441385e3660424d9228f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5852087793e042adad05275d4d232408" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3040a41717b441385e3660424d9228f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6046a148cd2749b2a3da0b64e879bc6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35399 } health_report { overall_health: UNKNOWN } } }
I20260927 02:41:12.744681 13661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:12.740894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44032 (local address 127.0.98.2:38381)
0927 02:41:12.741030 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:12.741035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:12.741052 (+ 17us) server_negotiation.cc:407] Connection header received
0927 02:41:12.741187 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:12.741190 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:12.741257 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:12.741362 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:41:12.741552 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:12.742349 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:12.743121 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:12.743460 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:12.743541 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:12.743778 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:41:12.744324 (+ 546us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:41:12.744428 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:12.744468 (+ 40us) server_negotiation.cc:299] Negotiation successful
0927 02:41:12.744525 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 02:41:12.750280 13316 raft_consensus.cc:1261] T b49dc530f36b434ea58fe042ac0f0e23 P 6046a148cd2749b2a3da0b64e879bc6a [term 1 FOLLOWER]: Refusing update from remote peer b3040a41717b441385e3660424d9228f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:41:12.750280 13584 raft_consensus.cc:1261] T b49dc530f36b434ea58fe042ac0f0e23 P 5852087793e042adad05275d4d232408 [term 1 FOLLOWER]: Refusing update from remote peer b3040a41717b441385e3660424d9228f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:41:12.751011 13684 consensus_queue.cc:1035] T b49dc530f36b434ea58fe042ac0f0e23 P b3040a41717b441385e3660424d9228f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6046a148cd2749b2a3da0b64e879bc6a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:41:12.751183 13672 consensus_queue.cc:1035] T b49dc530f36b434ea58fe042ac0f0e23 P b3040a41717b441385e3660424d9228f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5852087793e042adad05275d4d232408" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:41:12.759451 13687 mvcc.cc:205] Tried to move back new op lower bound from 7333793270778347520 to 7333793270434004992. Current Snapshot: MvccSnapshot[applied={T|T < 7333793270778347520}]
I20260927 02:41:12.760010 13689 mvcc.cc:205] Tried to move back new op lower bound from 7333793270778347520 to 7333793270434004992. Current Snapshot: MvccSnapshot[applied={T|T < 7333793270778347520}]
I20260927 02:41:12.766469 13688 mvcc.cc:205] Tried to move back new op lower bound from 7333793270778347520 to 7333793270434004992. Current Snapshot: MvccSnapshot[applied={T|T < 7333793270778347520 or (T in {7333793270778347520})}]
I20260927 02:41:12.767882 13164 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:41:12.768033 13164 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:41:12.770668 13164 catalog_manager.cc:5782] T 00000000000000000000000000000000 P acea6dc179644646b61d266863dde992: Sending DeleteTablet for 3 replicas of tablet b49dc530f36b434ea58fe042ac0f0e23
I20260927 02:41:12.771560 13564 tablet_service.cc:1514] Processing DeleteTablet for tablet b49dc530f36b434ea58fe042ac0f0e23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:41:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45434
I20260927 02:41:12.772209 13431 tablet_service.cc:1514] Processing DeleteTablet for tablet b49dc530f36b434ea58fe042ac0f0e23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:41:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44024
I20260927 02:41:12.772626 13697 tablet_replica.cc:330] T b49dc530f36b434ea58fe042ac0f0e23 P b3040a41717b441385e3660424d9228f: stopping tablet replica
I20260927 02:41:12.772905 13697 raft_consensus.cc:2229] T b49dc530f36b434ea58fe042ac0f0e23 P b3040a41717b441385e3660424d9228f [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:41:12.773187 13697 raft_consensus.cc:2258] T b49dc530f36b434ea58fe042ac0f0e23 P b3040a41717b441385e3660424d9228f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:41:12.773283 13296 tablet_service.cc:1514] Processing DeleteTablet for tablet b49dc530f36b434ea58fe042ac0f0e23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:41:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44040
I20260927 02:41:12.773804 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 13202
I20260927 02:41:12.774323 13697 ts_tablet_manager.cc:1905] T b49dc530f36b434ea58fe042ac0f0e23 P b3040a41717b441385e3660424d9228f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:41:12.776863 13697 ts_tablet_manager.cc:1918] T b49dc530f36b434ea58fe042ac0f0e23 P b3040a41717b441385e3660424d9228f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:41:12.776942 13697 log.cc:1196] T b49dc530f36b434ea58fe042ac0f0e23 P b3040a41717b441385e3660424d9228f: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-1/wal/wals/b49dc530f36b434ea58fe042ac0f0e23
I20260927 02:41:12.777213 13697 ts_tablet_manager.cc:1939] T b49dc530f36b434ea58fe042ac0f0e23 P b3040a41717b441385e3660424d9228f: Deleting consensus metadata
I20260927 02:41:12.778496 13148 catalog_manager.cc:4841] TS b3040a41717b441385e3660424d9228f (127.0.98.2:38381): tablet b49dc530f36b434ea58fe042ac0f0e23 (table test-table [id=2d2a2d9b5cc843b2937555acd9579c9b]) successfully deleted
I20260927 02:41:12.782469 13696 tablet_replica.cc:330] T b49dc530f36b434ea58fe042ac0f0e23 P 5852087793e042adad05275d4d232408: stopping tablet replica
I20260927 02:41:12.782799 13696 raft_consensus.cc:2229] T b49dc530f36b434ea58fe042ac0f0e23 P 5852087793e042adad05275d4d232408 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:41:12.783107 13696 raft_consensus.cc:2258] T b49dc530f36b434ea58fe042ac0f0e23 P 5852087793e042adad05275d4d232408 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 02:41:12.783345 13149 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35399 (error 108)
I20260927 02:41:12.784108 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 13366
I20260927 02:41:12.784938 13696 ts_tablet_manager.cc:1905] T b49dc530f36b434ea58fe042ac0f0e23 P 5852087793e042adad05275d4d232408: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:41:12.786001 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:12.785740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35399 (local address 127.0.0.1:44052)
0927 02:41:12.785868 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:12.785930 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 02:41:12.786191 13149 catalog_manager.cc:4584] TS 6046a148cd2749b2a3da0b64e879bc6a (127.0.98.1:35399): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b49dc530f36b434ea58fe042ac0f0e23: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35399: connect: Connection refused (error 111)
I20260927 02:41:12.787599 13696 ts_tablet_manager.cc:1918] T b49dc530f36b434ea58fe042ac0f0e23 P 5852087793e042adad05275d4d232408: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:41:12.787693 13696 log.cc:1196] T b49dc530f36b434ea58fe042ac0f0e23 P 5852087793e042adad05275d4d232408: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790476718023131-392-0/minicluster-data/ts-2/wal/wals/b49dc530f36b434ea58fe042ac0f0e23
I20260927 02:41:12.787974 13696 ts_tablet_manager.cc:1939] T b49dc530f36b434ea58fe042ac0f0e23 P 5852087793e042adad05275d4d232408: Deleting consensus metadata
I20260927 02:41:12.789240 13151 catalog_manager.cc:4841] TS 5852087793e042adad05275d4d232408 (127.0.98.3:36477): tablet b49dc530f36b434ea58fe042ac0f0e23 (table test-table [id=2d2a2d9b5cc843b2937555acd9579c9b]) successfully deleted
I20260927 02:41:12.790546 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 13500
I20260927 02:41:12.795318 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 13131
I20260927 02:41:12.802731 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 02:41:13.137027 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 02:41:13.137131 392 mini_ranger.cc:66] Stopping Ranger...
I20260927 02:41:13.471642 392 mini_ranger.cc:68] Stopped Ranger
2026-09-27 02:41:13.471 UTC [12198] LOG: received smart shutdown request
2026-09-27 02:41:13.475 UTC [12198] LOG: background worker "logical replication launcher" (PID 12207) exited with exit code 1
2026-09-27 02:41:13.475 UTC [12202] LOG: shutting down
2026-09-27 02:41:13.624 UTC [12198] LOG: database system is shut down
2026-09-27T02:41:13Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (46417 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:41:13 dist-test-slave-zv8z krb5kdc[13708](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:41:13 dist-test-slave-zv8z krb5kdc[13708](info): set up 2 sockets
Sep 27 02:41:13 dist-test-slave-zv8z krb5kdc[13708](info): commencing operation
krb5kdc: starting...
W20260927 02:41:14.690039 392 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/krb5kdc/test-user.keytab.
Sep 27 02:41:14 dist-test-slave-zv8z krb5kdc[13708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:41:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:41:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:41:14.803398 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35145
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35145,127.0.98.61:41151,127.0.98.60:40715
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:41:14.908939 13721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:41:14.909214 13721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:41:14.909269 13721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:41:14.913250 13721 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:41:14.913314 13721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:41:14.913341 13721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:41:14.913360 13721 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:41:14.913381 13721 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:41:14.918753 13721 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35145,127.0.98.61:41151,127.0.98.60:40715
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:41:14.919800 13721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:41:14.920817 13721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:41:14.929065 13726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:41:14.929069 13729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:41:14.929126 13721 server_base.cc:1034] running on GCE node
W20260927 02:41:14.929065 13727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:41:14.929945 13721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:41:14.931053 13721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:41:14.932214 13721 hybrid_clock.cc:648] HybridClock initialized: now 1790476874932191 us; error 34 us; skew 500 ppm
Sep 27 02:41:14 dist-test-slave-zv8z krb5kdc[13708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:41:14.934777 13721 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:41:14.935621 13721 webserver.cc:460] Webserver started at http://127.0.98.62:43373/ using document root <none> and password file <none>
I20260927 02:41:14.936108 13721 fs_manager.cc:359] Metadata directory not provided
I20260927 02:41:14.936172 13721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:41:14.936328 13721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:41:14.938235 13721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/minicluster-data/master-0/data/instance:
uuid: "51aec98a76574a3cadaf615315bbbb0b"
format_stamp: "Formatted at 2026-09-27 02:41:14 on dist-test-slave-zv8z"
server_key: "f9299036c0c919f4ba3bd114a3ac5c43"
server_key_iv: "c099856793e5950f700af480403df4bb"
server_key_version: "encryptionkey@0"
I20260927 02:41:14.938791 13721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/minicluster-data/master-0/wal/instance:
uuid: "51aec98a76574a3cadaf615315bbbb0b"
format_stamp: "Formatted at 2026-09-27 02:41:14 on dist-test-slave-zv8z"
server_key: "f9299036c0c919f4ba3bd114a3ac5c43"
server_key_iv: "c099856793e5950f700af480403df4bb"
server_key_version: "encryptionkey@0"
I20260927 02:41:14.942348 13721 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260927 02:41:14.944634 13736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:14.945812 13721 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:41:14.945909 13721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "51aec98a76574a3cadaf615315bbbb0b"
format_stamp: "Formatted at 2026-09-27 02:41:14 on dist-test-slave-zv8z"
server_key: "f9299036c0c919f4ba3bd114a3ac5c43"
server_key_iv: "c099856793e5950f700af480403df4bb"
server_key_version: "encryptionkey@0"
I20260927 02:41:14.945986 13721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:41:14.970304 13721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:41:14.973008 13721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:41:14.973168 13721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:41:14.979806 13721 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35145
I20260927 02:41:14.979830 13788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35145 every 8 connection(s)
I20260927 02:41:14.980708 13721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:41:14.983412 13789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:41:14.987056 13789 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35145 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:41:14.989460 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 13721
I20260927 02:41:14.989607 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/minicluster-data/master-0/wal/instance
I20260927 02:41:14.989916 392 external_mini_cluster.cc:1428] Setting key d303ba1ceae333de9011fb3e89867669
Sep 27 02:41:14 dist-test-slave-zv8z krb5kdc[13708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260927 02:41:15.002372 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:14.989223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35145 (local address 127.0.0.1:42336)
0927 02:41:14.989845 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:14.989891 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 02:41:14.990767 (+ 876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:41:14.991531 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:41:14.991537 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:41:14.991996 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:41:14.992334 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:14.992352 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:14.993963 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:14.993966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:14.994751 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:14.994771 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:14.995659 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:14.995662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:14.995756 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:14.996394 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:41:14.996410 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:41:14.997743 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:41:14.999630 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:14.999638 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:14.999646 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:14.999876 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:15.000197 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:15.000200 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:15.000202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:15.000337 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:15.000878 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:41:15.000885 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:41:15.001059 (+ 174us) client_negotiation.cc:769] Sending connection context
0927 02:41:15.001243 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 02:41:15.001498 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":138,"threads_started":1}
I20260927 02:41:15.002373 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:14.989233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42336 (local address 127.0.98.62:35145)
0927 02:41:14.989688 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:14.989708 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:14.990047 (+ 339us) server_negotiation.cc:407] Connection header received
0927 02:41:14.991069 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:14.991075 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:14.991321 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:14.991497 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:41:14.992501 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:14.993826 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:14.994880 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:14.995464 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:14.995624 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:14.997874 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:41:14.997880 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:41:14.997890 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:41:14.997911 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:41:14.999519 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:14.999973 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:14.999978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:14.999979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:15.000093 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:15.000415 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:15.000417 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:15.000418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:15.000771 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:41:15.000874 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:15.001352 (+ 478us) server_negotiation.cc:299] Negotiation successful
0927 02:41:15.001619 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":310,"spinlock_wait_cycles":50944,"thread_start_us":138,"threads_started":1}
I20260927 02:41:15.004710 13789 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41151 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:41:15.005414 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:15.005172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41151 (local address 127.0.0.1:50484)
0927 02:41:15.005285 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:15.005354 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 02:41:15.005609 13738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41151: connect: Connection refused (error 111)
I20260927 02:41:15.006434 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:15.006228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41151 (local address 127.0.0.1:50488)
0927 02:41:15.006342 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:15.006385 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 02:41:15.006561 13789 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41151: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41151: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 02:41:15.017534 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41151
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35145,127.0.98.61:41151,127.0.98.60:40715
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:41:15.055794 13789 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41151 } attempt: 1
I20260927 02:41:15.056646 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:15.056356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41151 (local address 127.0.0.1:50498)
0927 02:41:15.056502 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:15.056566 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 02:41:15.057294 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:15.057089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41151 (local address 127.0.0.1:50502)
0927 02:41:15.057195 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:15.057238 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 02:41:15.057402 13789 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41151: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41151: connect: Connection refused (error 111)
I20260927 02:41:15.125559 13789 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41151 } attempt: 2
I20260927 02:41:15.126454 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:15.126106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41151 (local address 127.0.0.1:50514)
0927 02:41:15.126292 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:15.126352 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 02:41:15.127188 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:15.126971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41151 (local address 127.0.0.1:50518)
0927 02:41:15.127078 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:15.127123 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 02:41:15.127313 13789 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41151: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41151: connect: Connection refused (error 111)
W20260927 02:41:15.127919 13796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:41:15.128163 13796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:41:15.128232 13796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:41:15.132205 13796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:41:15.132254 13796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:41:15.132279 13796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:41:15.132302 13796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:41:15.132330 13796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:41:15.137564 13796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35145,127.0.98.61:41151,127.0.98.60:40715
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:41:15.138626 13796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:41:15.139600 13796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:41:15.148126 13801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:41:15.148180 13802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:41:15.148206 13796 server_base.cc:1034] running on GCE node
W20260927 02:41:15.148280 13804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:41:15.148864 13796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:41:15.149951 13796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:41:15.151106 13796 hybrid_clock.cc:648] HybridClock initialized: now 1790476875151066 us; error 52 us; skew 500 ppm
Sep 27 02:41:15 dist-test-slave-zv8z krb5kdc[13708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:41:15.153724 13796 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 02:41:15.154580 13796 webserver.cc:460] Webserver started at http://127.0.98.61:36051/ using document root <none> and password file <none>
I20260927 02:41:15.155093 13796 fs_manager.cc:359] Metadata directory not provided
I20260927 02:41:15.155156 13796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:41:15.155339 13796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:41:15.157271 13796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/minicluster-data/master-1/data/instance:
uuid: "d3f339de7597470197c4cc7ce9e88f11"
format_stamp: "Formatted at 2026-09-27 02:41:15 on dist-test-slave-zv8z"
server_key: "aebcc10044acbf286c0f02bd1f240b1a"
server_key_iv: "51ce01a8a342fd8308e1faaad27c49f6"
server_key_version: "encryptionkey@0"
I20260927 02:41:15.157793 13796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/minicluster-data/master-1/wal/instance:
uuid: "d3f339de7597470197c4cc7ce9e88f11"
format_stamp: "Formatted at 2026-09-27 02:41:15 on dist-test-slave-zv8z"
server_key: "aebcc10044acbf286c0f02bd1f240b1a"
server_key_iv: "51ce01a8a342fd8308e1faaad27c49f6"
server_key_version: "encryptionkey@0"
I20260927 02:41:15.161381 13796 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 02:41:15.163815 13811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:15.165102 13796 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:41:15.165210 13796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/minicluster-data/master-1/wal
uuid: "d3f339de7597470197c4cc7ce9e88f11"
format_stamp: "Formatted at 2026-09-27 02:41:15 on dist-test-slave-zv8z"
server_key: "aebcc10044acbf286c0f02bd1f240b1a"
server_key_iv: "51ce01a8a342fd8308e1faaad27c49f6"
server_key_version: "encryptionkey@0"
I20260927 02:41:15.165298 13796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:41:15.187137 13796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:41:15.190137 13796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:41:15.190332 13796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:41:15.196852 13796 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:41151
I20260927 02:41:15.196897 13863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:41151 every 8 connection(s)
I20260927 02:41:15.197777 13796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/minicluster-data/master-1/data/info.pb
I20260927 02:41:15.200490 13864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:41:15.203347 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 13796
I20260927 02:41:15.203454 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/minicluster-data/master-1/wal/instance
I20260927 02:41:15.203718 392 external_mini_cluster.cc:1428] Setting key 8496eb2a6e86950246252897350e2130
I20260927 02:41:15.204147 13864 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35145 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 27 02:41:15 dist-test-slave-zv8z krb5kdc[13708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:41:15.218453 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:15.206184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42350 (local address 127.0.98.62:35145)
0927 02:41:15.206356 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:15.206361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:15.207186 (+ 825us) server_negotiation.cc:407] Connection header received
0927 02:41:15.208145 (+ 959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:15.208149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:15.208210 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:15.208319 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:41:15.209427 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:15.210235 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:15.211393 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:15.211652 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:15.211736 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:15.214217 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:41:15.214220 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:41:15.214222 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:41:15.214269 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:41:15.215771 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:15.216258 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:15.216262 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:15.216264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:15.216348 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:15.216625 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:15.216628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:15.216630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:15.216851 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:41:15.216952 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:15.218119 (+ 1167us) server_negotiation.cc:299] Negotiation successful
0927 02:41:15.218279 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 02:41:15.218632 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:15.206572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35145 (local address 127.0.0.1:42350)
0927 02:41:15.207027 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:15.207078 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 02:41:15.207921 (+ 843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:41:15.208445 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:41:15.208452 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:41:15.208798 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:41:15.209277 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:15.209290 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:15.210378 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:15.210381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:15.211283 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:15.211293 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:15.211782 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:15.211785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:15.211926 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:15.212453 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:41:15.212494 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:41:15.214027 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:41:15.215882 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:15.215887 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:15.215899 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:15.216166 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:15.216424 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:15.216425 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:15.216427 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:15.216542 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:15.217000 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:41:15.217005 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:41:15.217266 (+ 261us) client_negotiation.cc:769] Sending connection context
0927 02:41:15.217437 (+ 171us) client_negotiation.cc:240] Negotiation successful
0927 02:41:15.217709 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":119,"threads_started":1}
I20260927 02:41:15.218452 13789 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41151 } attempt: 3
I20260927 02:41:15.220355 13864 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41151 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:41:15 dist-test-slave-zv8z krb5kdc[13708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 02:41:15 dist-test-slave-zv8z krb5kdc[13708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 02:41:15.228760 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:15.219326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41151 (local address 127.0.0.1:50534)
0927 02:41:15.219499 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:15.219514 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:41:15.219596 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:41:15.220144 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:41:15.220147 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:41:15.220300 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:41:15.220424 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:15.220431 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:15.221635 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:15.221638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:15.222234 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:15.222242 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:15.222955 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:15.222959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:15.223067 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:15.223541 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:41:15.223552 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:41:15.224965 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:41:15.227284 (+ 2319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:15.227288 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:15.227290 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:15.227492 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:15.227757 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:15.227760 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:15.227762 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:15.227844 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:15.228270 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:41:15.228273 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:41:15.228368 (+ 95us) client_negotiation.cc:769] Sending connection context
0927 02:41:15.228465 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 02:41:15.228594 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260927 02:41:15.228864 13869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:15.219414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50534 (local address 127.0.98.61:41151)
0927 02:41:15.219757 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:15.219768 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:15.219786 (+ 18us) server_negotiation.cc:407] Connection header received
0927 02:41:15.219823 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:15.219828 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:15.220006 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:15.220136 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:41:15.220569 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:15.221511 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:15.222409 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:15.222838 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:15.222941 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:15.225095 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:41:15.225105 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:41:15.225119 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:41:15.225143 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:41:15.227188 (+ 2045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:15.227582 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:15.227585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:15.227586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:15.227678 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:15.227912 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:15.227916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:15.227917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:15.228172 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:41:15.228284 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:15.228575 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 02:41:15.228707 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":256,"thread_start_us":99,"threads_started":1}
I20260927 02:41:15.229094 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:15.220802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41151 (local address 127.0.0.1:50540)
0927 02:41:15.220894 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:15.220908 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:41:15.220975 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:41:15.221439 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:41:15.221441 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:41:15.221591 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:41:15.221686 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:15.221692 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:15.222487 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:15.222489 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:15.223179 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:15.223189 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:15.223715 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:15.223719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:15.223812 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:15.224243 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:41:15.224254 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:41:15.225403 (+ 1149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:41:15.227250 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:15.227253 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:15.227255 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:15.227476 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:15.227731 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:15.227733 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:15.227735 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:15.227821 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:15.228629 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:41:15.228632 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:41:15.228735 (+ 103us) client_negotiation.cc:769] Sending connection context
0927 02:41:15.228813 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 02:41:15.228936 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260927 02:41:15.229183 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40715
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35145,127.0.98.61:41151,127.0.98.60:40715
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:41:15.229666 13870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:15.220957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50540 (local address 127.0.98.61:41151)
0927 02:41:15.221257 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:15.221263 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:15.221273 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:41:15.221298 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:15.221300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:15.221330 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:15.221405 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:41:15.221790 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:15.222379 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:15.223293 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:15.223608 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:15.223683 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:15.225541 (+ 1858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:41:15.225544 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:41:15.225547 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:41:15.225567 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:41:15.227161 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:15.227570 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:15.227575 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:15.227576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:15.227654 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:15.227916 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:15.227919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:15.227921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:15.228176 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:41:15.229368 (+ 1192us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:15.229434 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 02:41:15.229526 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":14,"server-negotiator.queue_time_us":252,"spinlock_wait_cycles":1408,"thread_start_us":144,"threads_started":1}
I20260927 02:41:15.231011 13789 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40715 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:41:15.231098 13864 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40715 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:41:15.231706 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:15.231487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40715 (local address 127.0.0.1:42064)
0927 02:41:15.231603 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:15.231650 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 02:41:15.231961 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:15.231741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40715 (local address 127.0.0.1:42070)
0927 02:41:15.231823 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:15.231902 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260927 02:41:15.232125 13812 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40715: connect: Connection refused (error 111)
I20260927 02:41:15.232378 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:15.232123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40715 (local address 127.0.0.1:42086)
0927 02:41:15.232279 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:15.232318 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260927 02:41:15.232582 13789 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40715: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40715: connect: Connection refused (error 111)
I20260927 02:41:15.232865 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:15.232673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40715 (local address 127.0.0.1:42092)
0927 02:41:15.232787 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:15.232820 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260927 02:41:15.232985 13864 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40715: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40715: connect: Connection refused (error 111)
I20260927 02:41:15.263739 13789 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40715 } attempt: 1
I20260927 02:41:15.264639 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:15.264299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40715 (local address 127.0.0.1:42108)
0927 02:41:15.264467 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:15.264549 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 02:41:15.265334 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:15.265111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40715 (local address 127.0.0.1:42124)
0927 02:41:15.265216 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:15.265268 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 02:41:15.265455 13789 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40715: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40715: connect: Connection refused (error 111)
I20260927 02:41:15.282215 13864 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40715 } attempt: 1
I20260927 02:41:15.283006 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:15.282733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40715 (local address 127.0.0.1:42132)
0927 02:41:15.282868 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:15.282929 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 02:41:15.283756 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:15.283562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40715 (local address 127.0.0.1:42136)
0927 02:41:15.283657 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:15.283700 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 02:41:15.283864 13864 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40715: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40715: connect: Connection refused (error 111)
W20260927 02:41:15.336102 13871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:41:15.336340 13871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:41:15.336377 13871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:41:15.340309 13871 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:41:15.340361 13871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:41:15.340379 13871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:41:15.340394 13871 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:41:15.340415 13871 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:41:15.340600 13789 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40715 } attempt: 2
I20260927 02:41:15.341396 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:15.341085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40715 (local address 127.0.0.1:42152)
0927 02:41:15.341242 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:15.341310 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 02:41:15.342057 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:15.341844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40715 (local address 127.0.0.1:42158)
0927 02:41:15.341950 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:15.341999 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 02:41:15.342227 13789 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40715: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40715: connect: Connection refused (error 111)
I20260927 02:41:15.345865 13871 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35145,127.0.98.61:41151,127.0.98.60:40715
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:41:15.346872 13871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:41:15.347826 13871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:41:15.351981 13864 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40715 } attempt: 2
I20260927 02:41:15.352696 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:15.352426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40715 (local address 127.0.0.1:42172)
0927 02:41:15.352572 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:15.352625 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 02:41:15.353356 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:15.353160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40715 (local address 127.0.0.1:42174)
0927 02:41:15.353260 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:15.353302 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 02:41:15.353459 13864 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40715: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40715: connect: Connection refused (error 111)
W20260927 02:41:15.356290 13880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:41:15.356328 13878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:41:15.356308 13877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:41:15.356400 13871 server_base.cc:1034] running on GCE node
I20260927 02:41:15.357131 13871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:41:15.358220 13871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:41:15.359419 13871 hybrid_clock.cc:648] HybridClock initialized: now 1790476875359400 us; error 29 us; skew 500 ppm
Sep 27 02:41:15 dist-test-slave-zv8z krb5kdc[13708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:41:15.362203 13871 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 02:41:15.363085 13871 webserver.cc:460] Webserver started at http://127.0.98.60:45411/ using document root <none> and password file <none>
I20260927 02:41:15.363598 13871 fs_manager.cc:359] Metadata directory not provided
I20260927 02:41:15.363662 13871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:41:15.363847 13871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:41:15.365818 13871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/minicluster-data/master-2/data/instance:
uuid: "42ee57ed2af04a7687f89fe8549d1dc4"
format_stamp: "Formatted at 2026-09-27 02:41:15 on dist-test-slave-zv8z"
server_key: "a0b3ca49821ffa4204008b6dc8f30a71"
server_key_iv: "4b712931e1fcd0cc10747bffd97763a0"
server_key_version: "encryptionkey@0"
I20260927 02:41:15.366359 13871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/minicluster-data/master-2/wal/instance:
uuid: "42ee57ed2af04a7687f89fe8549d1dc4"
format_stamp: "Formatted at 2026-09-27 02:41:15 on dist-test-slave-zv8z"
server_key: "a0b3ca49821ffa4204008b6dc8f30a71"
server_key_iv: "4b712931e1fcd0cc10747bffd97763a0"
server_key_version: "encryptionkey@0"
I20260927 02:41:15.369715 13871 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:41:15.371944 13887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:15.373039 13871 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:41:15.373143 13871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/minicluster-data/master-2/wal
uuid: "42ee57ed2af04a7687f89fe8549d1dc4"
format_stamp: "Formatted at 2026-09-27 02:41:15 on dist-test-slave-zv8z"
server_key: "a0b3ca49821ffa4204008b6dc8f30a71"
server_key_iv: "4b712931e1fcd0cc10747bffd97763a0"
server_key_version: "encryptionkey@0"
I20260927 02:41:15.373224 13871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:41:15.393512 13871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:41:15.396556 13871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:41:15.396728 13871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:41:15.403527 13871 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:40715
I20260927 02:41:15.403568 13939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:40715 every 8 connection(s)
I20260927 02:41:15.404419 13871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/minicluster-data/master-2/data/info.pb
I20260927 02:41:15.406023 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 13871
I20260927 02:41:15.406143 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790476718023131-392-0/minicluster-data/master-2/wal/instance
I20260927 02:41:15.406448 392 external_mini_cluster.cc:1428] Setting key 8a99e063a835d0682e2aa147e2d9205b
I20260927 02:41:15.407238 13940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:41:15.411157 13940 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35145 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:41:15 dist-test-slave-zv8z krb5kdc[13708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 02:41:15 dist-test-slave-zv8z krb5kdc[13708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:41:15.420110 13943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:15.407663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42178 (local address 127.0.98.60:40715)
0927 02:41:15.408031 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:15.408046 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:15.408080 (+ 34us) server_negotiation.cc:407] Connection header received
0927 02:41:15.408687 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:15.408703 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:15.409020 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:15.409281 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:41:15.410137 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:15.411141 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:15.412311 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:15.412774 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:15.412928 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:15.415214 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:41:15.415221 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:41:15.415232 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:41:15.415262 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:41:15.417320 (+ 2058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:15.417760 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:15.417765 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:15.417769 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:15.417872 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:15.418174 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:15.418178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:15.418180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:15.418584 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:41:15.418746 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:15.419079 (+ 333us) server_negotiation.cc:299] Negotiation successful
0927 02:41:15.419341 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":100,"threads_started":1}
I20260927 02:41:15.421602 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:15.412969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42356 (local address 127.0.98.62:35145)
0927 02:41:15.413091 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:15.413096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:15.413430 (+ 334us) server_negotiation.cc:407] Connection header received
0927 02:41:15.413589 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:15.413592 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:15.413637 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:15.413703 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:41:15.414340 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:15.414852 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:15.415868 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:15.416176 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:15.416276 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:15.418584 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:41:15.418587 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:41:15.418589 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:41:15.418613 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:41:15.419927 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:15.420380 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:15.420384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:15.420385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:15.420481 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:15.420715 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:15.420717 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:15.420719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:15.420956 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:41:15.421049 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:15.421354 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 02:41:15.421460 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 02:41:15.421622 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:15.412869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35145 (local address 127.0.0.1:42356)
0927 02:41:15.413302 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:15.413350 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 02:41:15.413474 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:41:15.413757 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:41:15.413763 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:41:15.414099 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:41:15.414214 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:15.414220 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:15.414961 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:15.414964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:15.415753 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:15.415764 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:15.416287 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:15.416291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:15.416403 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:15.416810 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:41:15.416824 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:41:15.418460 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:41:15.420030 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:15.420040 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:15.420046 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:15.420262 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:15.420568 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:15.420570 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:15.420571 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:15.420640 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:15.421061 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:41:15.421066 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:41:15.421160 (+ 94us) client_negotiation.cc:769] Sending connection context
0927 02:41:15.421274 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 02:41:15.421433 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":125,"threads_started":1}
I20260927 02:41:15.422747 13940 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41151 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:41:15 dist-test-slave-zv8z krb5kdc[13708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 02:41:15 dist-test-slave-zv8z krb5kdc[13708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 02:41:15.430510 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:15.422423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42358 (local address 127.0.98.62:35145)
0927 02:41:15.422581 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:15.422584 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:15.422595 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:41:15.422668 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:15.422671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:15.422711 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:15.422788 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:41:15.423418 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:15.424013 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:15.424861 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:15.425061 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:15.425128 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:15.427523 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:41:15.427526 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:41:15.427528 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:41:15.427548 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:41:15.428967 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:15.429368 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:15.429370 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:15.429371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:15.429447 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:15.429715 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:15.429718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:15.429720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:15.429949 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:41:15.430164 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:15.430239 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 02:41:15.430368 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 02:41:15.430655 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:15.423187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41151 (local address 127.0.0.1:50546)
0927 02:41:15.423327 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:15.423339 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:41:15.423397 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:41:15.423665 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:41:15.423667 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:41:15.423787 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:41:15.423872 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:15.423877 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:15.424679 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:15.424683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:15.425307 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:15.425317 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:15.425956 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:15.425960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:15.426058 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:15.426455 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:41:15.426469 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:41:15.427691 (+ 1222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:41:15.429323 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:15.429326 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:15.429328 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:15.429504 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:15.429755 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:15.429757 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:15.429759 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:15.429825 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:15.430198 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:41:15.430201 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:41:15.430311 (+ 110us) client_negotiation.cc:769] Sending connection context
0927 02:41:15.430410 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 02:41:15.430517 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260927 02:41:15.430722 13870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:15.423340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50546 (local address 127.0.98.61:41151)
0927 02:41:15.423453 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:15.423457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:15.423469 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:41:15.423507 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:15.423509 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:15.423578 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:15.423657 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:41:15.423961 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:15.424586 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:15.425443 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:15.425782 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:15.425877 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:15.427810 (+ 1933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:41:15.427814 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:41:15.427817 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:41:15.427854 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:41:15.429223 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:15.429596 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:15.429599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:15.429600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:15.429668 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:15.429906 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:15.429909 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:15.429911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:15.430103 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:41:15.430191 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:15.430475 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 02:41:15.430583 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 02:41:15.431279 13940 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40715 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:41:15 dist-test-slave-zv8z krb5kdc[13708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 02:41:15 dist-test-slave-zv8z krb5kdc[13708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 02:41:15.438742 13943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:15.431755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42184 (local address 127.0.98.60:40715)
0927 02:41:15.431838 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:15.431841 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:15.431853 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:41:15.431962 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:15.431965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:15.431998 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:15.432040 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:41:15.432470 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:15.433098 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:15.433863 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:15.434179 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:15.434290 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:15.436191 (+ 1901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:41:15.436194 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:41:15.436197 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:41:15.436218 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:41:15.437423 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:15.437764 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:15.437767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:15.437769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:15.437832 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:15.438039 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:15.438042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:15.438044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:15.438193 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:41:15.438293 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:15.438534 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 02:41:15.438624 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":36}
I20260927 02:41:15.438745 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:15.431704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40715 (local address 127.0.0.1:42184)
0927 02:41:15.431789 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:15.431802 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:41:15.431869 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:41:15.432143 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:41:15.432146 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:41:15.432293 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:41:15.432366 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:15.432371 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:15.433195 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:15.433199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:15.433768 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:15.433776 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:15.434296 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:15.434300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:15.434402 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:15.434833 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:41:15.434843 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:41:15.436078 (+ 1235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:41:15.437514 (+ 1436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:15.437517 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:15.437519 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:15.437685 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:15.437902 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:15.437905 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:15.437907 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:15.437970 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:15.438290 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:41:15.438293 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:41:15.438369 (+ 76us) client_negotiation.cc:769] Sending connection context
0927 02:41:15.438465 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 02:41:15.438576 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":17}
I20260927 02:41:15.439572 13870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:15.431964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50556 (local address 127.0.98.61:41151)
0927 02:41:15.432062 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:15.432065 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:15.432076 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:41:15.432143 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:15.432146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:15.432181 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:15.432252 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:41:15.432810 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:15.433336 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:15.434118 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:15.434412 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:15.434508 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:15.436688 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:41:15.436691 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:41:15.436693 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:41:15.436713 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:41:15.438002 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:15.438398 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:15.438401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:15.438403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:15.438481 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:15.438769 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:15.438772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:15.438774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:15.439058 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:41:15.439190 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:15.439378 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 02:41:15.439456 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 02:41:15.441413 13789 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40715 } attempt: 3
I20260927 02:41:15.442520 13940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42ee57ed2af04a7687f89fe8549d1dc4: Bootstrap starting.
I20260927 02:41:15.442816 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:41:15.444746 13864 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40715 } attempt: 3
I20260927 02:41:15.445317 13940 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42ee57ed2af04a7687f89fe8549d1dc4: Neither blocks nor log segments found. Creating new log.
Sep 27 02:41:15 dist-test-slave-zv8z krb5kdc[13708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 02:41:15.446180 13940 log.cc:824] T 00000000000000000000000000000000 P 42ee57ed2af04a7687f89fe8549d1dc4: Log is configured to *not* fsync() on all Append() calls
I20260927 02:41:15.448237 13940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42ee57ed2af04a7687f89fe8549d1dc4: No bootstrap required, opened a new log
I20260927 02:41:15.449268 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:15.441788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40715 (local address 127.0.0.1:42186)
0927 02:41:15.441889 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:15.441903 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:41:15.441976 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:41:15.442180 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:41:15.442182 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:41:15.442360 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:41:15.442434 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:15.442439 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:15.443219 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:15.443222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:15.443720 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:15.443727 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:15.444216 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:15.444220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:15.444295 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:15.444685 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:41:15.444699 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:41:15.446022 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:41:15.447792 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:15.447794 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:15.447796 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:15.448025 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:15.448312 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:15.448315 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:15.448317 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:15.448397 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:15.448820 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:41:15.448823 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:41:15.448904 (+ 81us) client_negotiation.cc:769] Sending connection context
0927 02:41:15.448991 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 02:41:15.449110 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 02:41:15.449754 13943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:15.441841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42186 (local address 127.0.98.60:40715)
0927 02:41:15.441928 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:15.441931 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:15.441943 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:41:15.442060 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:15.442062 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:15.442096 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:15.442155 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:41:15.442533 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:15.443125 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:15.443817 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:15.444075 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:15.444464 (+ 389us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:15.446157 (+ 1693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:41:15.446161 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:41:15.446163 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:41:15.446186 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:41:15.447694 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:15.448135 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:15.448138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:15.448140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:15.448218 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:15.448509 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:15.448512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:15.448514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:15.448697 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:41:15.448780 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:15.449522 (+ 742us) server_negotiation.cc:299] Negotiation successful
0927 02:41:15.449624 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
Sep 27 02:41:15 dist-test-slave-zv8z krb5kdc[13708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 02:41:15.452212 13940 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42ee57ed2af04a7687f89fe8549d1dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51aec98a76574a3cadaf615315bbbb0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35145 } } peers { permanent_uuid: "d3f339de7597470197c4cc7ce9e88f11" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41151 } } peers { permanent_uuid: "42ee57ed2af04a7687f89fe8549d1dc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40715 } }
I20260927 02:41:15.452443 13940 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42ee57ed2af04a7687f89fe8549d1dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:41:15.452494 13940 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42ee57ed2af04a7687f89fe8549d1dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42ee57ed2af04a7687f89fe8549d1dc4, State: Initialized, Role: FOLLOWER
I20260927 02:41:15.452585 13789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51aec98a76574a3cadaf615315bbbb0b: Bootstrap starting.
I20260927 02:41:15.453415 13940 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42ee57ed2af04a7687f89fe8549d1dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51aec98a76574a3cadaf615315bbbb0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35145 } } peers { permanent_uuid: "d3f339de7597470197c4cc7ce9e88f11" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41151 } } peers { permanent_uuid: "42ee57ed2af04a7687f89fe8549d1dc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40715 } }
I20260927 02:41:15.454872 13940 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42ee57ed2af04a7687f89fe8549d1dc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:41:15.454918 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:15.445103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40715 (local address 127.0.0.1:42188)
0927 02:41:15.445353 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:15.445366 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:41:15.445434 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:41:15.446146 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:41:15.446149 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:41:15.446349 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:41:15.446440 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:15.446446 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:15.447466 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:15.447469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:15.448184 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:15.448194 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:15.448774 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:15.448778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:15.448853 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:15.449255 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:41:15.449268 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:41:15.450693 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:41:15.453447 (+ 2754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:15.453450 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:15.453452 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:15.453698 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:15.453992 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:15.453995 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:15.453997 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:15.454079 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:15.454485 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:41:15.454488 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:41:15.454590 (+ 102us) client_negotiation.cc:769] Sending connection context
0927 02:41:15.454665 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 02:41:15.454779 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185}
I20260927 02:41:15.455323 13789 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51aec98a76574a3cadaf615315bbbb0b: Neither blocks nor log segments found. Creating new log.
I20260927 02:41:15.455477 13950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:15.445625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42188 (local address 127.0.98.60:40715)
0927 02:41:15.445923 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:15.445934 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:15.445947 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:41:15.445979 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:15.445981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:15.446017 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:15.446558 (+ 541us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:41:15.446653 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:15.447345 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:15.448319 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:15.448650 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:15.449068 (+ 418us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:15.450830 (+ 1762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:41:15.450833 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:41:15.450836 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:41:15.450857 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:41:15.453280 (+ 2423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:15.453815 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:15.453817 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:15.453819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:15.453898 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:15.454177 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:15.454180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:15.454182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:15.454379 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:41:15.455155 (+ 776us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:15.455226 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 02:41:15.455325 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":89,"threads_started":1}
I20260927 02:41:15.456126 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:15.444873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42374 (local address 127.0.98.62:35145)
0927 02:41:15.445840 (+ 967us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:15.445844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:15.447463 (+ 1619us) server_negotiation.cc:407] Connection header received
0927 02:41:15.447627 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:15.447630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:15.447671 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:15.447716 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:41:15.448551 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:15.449497 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:15.450564 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:15.450839 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:15.450904 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:15.452862 (+ 1958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:41:15.452866 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:41:15.452868 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:41:15.452890 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:41:15.454410 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:15.454856 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:15.454859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:15.454861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:15.454935 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:15.455250 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:15.455254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:15.455255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:15.455444 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:41:15.455543 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:15.455881 (+ 338us) server_negotiation.cc:299] Negotiation successful
0927 02:41:15.455980 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":898}
I20260927 02:41:15.456203 13789 log.cc:824] T 00000000000000000000000000000000 P 51aec98a76574a3cadaf615315bbbb0b: Log is configured to *not* fsync() on all Append() calls
I20260927 02:41:15.458204 13864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3f339de7597470197c4cc7ce9e88f11: Bootstrap starting.
I20260927 02:41:15.459367 13789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51aec98a76574a3cadaf615315bbbb0b: No bootstrap required, opened a new log
I20260927 02:41:15.460609 13864 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3f339de7597470197c4cc7ce9e88f11: Neither blocks nor log segments found. Creating new log.
I20260927 02:41:15.461483 13864 log.cc:824] T 00000000000000000000000000000000 P d3f339de7597470197c4cc7ce9e88f11: Log is configured to *not* fsync() on all Append() calls
I20260927 02:41:15.463191 13789 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51aec98a76574a3cadaf615315bbbb0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51aec98a76574a3cadaf615315bbbb0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35145 } } peers { permanent_uuid: "d3f339de7597470197c4cc7ce9e88f11" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41151 } } peers { permanent_uuid: "42ee57ed2af04a7687f89fe8549d1dc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40715 } }
I20260927 02:41:15.459088 13871 master_runner.cc:186] Error getting master registration for 127.0.98.62:35145: OK, instance_id { permanent_uuid: "51aec98a76574a3cadaf615315bbbb0b" instance_seqno: 1790476874977930 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 02:41:15.463371 13789 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51aec98a76574a3cadaf615315bbbb0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:41:15.463438 13789 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51aec98a76574a3cadaf615315bbbb0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51aec98a76574a3cadaf615315bbbb0b, State: Initialized, Role: FOLLOWER
I20260927 02:41:15.463469 13871 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 02:41:15.463873 13789 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51aec98a76574a3cadaf615315bbbb0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51aec98a76574a3cadaf615315bbbb0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35145 } } peers { permanent_uuid: "d3f339de7597470197c4cc7ce9e88f11" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41151 } } peers { permanent_uuid: "42ee57ed2af04a7687f89fe8549d1dc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40715 } }
I20260927 02:41:15.465281 13789 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51aec98a76574a3cadaf615315bbbb0b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:41:15.466292 13870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:15.445119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50558 (local address 127.0.98.61:41151)
0927 02:41:15.445554 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:15.445558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:15.447023 (+ 1465us) server_negotiation.cc:407] Connection header received
0927 02:41:15.456689 (+ 9666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:15.456692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:15.456735 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:15.456784 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:41:15.457614 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:15.458265 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:15.459247 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:15.459574 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:15.459637 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:15.461227 (+ 1590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:41:15.461231 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:41:15.461234 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:41:15.461257 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:41:15.464085 (+ 2828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:15.464509 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:15.464512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:15.464514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:15.464819 (+ 305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:15.465142 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:15.465145 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:15.465147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:15.465340 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:41:15.465436 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:15.465776 (+ 340us) server_negotiation.cc:299] Negotiation successful
0927 02:41:15.465880 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381}
W20260927 02:41:15.467090 13967 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 42ee57ed2af04a7687f89fe8549d1dc4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:41:15.467200 13967 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:41:15.468133 13955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42ee57ed2af04a7687f89fe8549d1dc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51aec98a76574a3cadaf615315bbbb0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35145 } } peers { permanent_uuid: "d3f339de7597470197c4cc7ce9e88f11" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41151 } } peers { permanent_uuid: "42ee57ed2af04a7687f89fe8549d1dc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40715 } } }
I20260927 02:41:15.468282 13955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42ee57ed2af04a7687f89fe8549d1dc4 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:41:15.468501 13969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51aec98a76574a3cadaf615315bbbb0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51aec98a76574a3cadaf615315bbbb0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35145 } } peers { permanent_uuid: "d3f339de7597470197c4cc7ce9e88f11" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41151 } } peers { permanent_uuid: "42ee57ed2af04a7687f89fe8549d1dc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40715 } } }
I20260927 02:41:15.468868 13969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51aec98a76574a3cadaf615315bbbb0b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:41:15.469746 13952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:15.446309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42198 (local address 127.0.98.60:40715)
0927 02:41:15.447256 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:15.447263 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:15.451585 (+ 4322us) server_negotiation.cc:407] Connection header received
0927 02:41:15.451774 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:15.451777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:15.451815 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:15.451866 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:41:15.461922 (+ 10056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:15.462777 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:15.463803 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:15.464147 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:15.464242 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:15.465648 (+ 1406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:41:15.465652 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:41:15.465654 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:41:15.465678 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:41:15.467282 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:15.468148 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:15.468151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:15.468153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:15.468234 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:15.468511 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:15.468514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:15.468516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:15.468711 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:41:15.468825 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:15.469463 (+ 638us) server_negotiation.cc:299] Negotiation successful
0927 02:41:15.469572 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":886,"thread_start_us":314,"threads_started":1}
I20260927 02:41:15.470887 13864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3f339de7597470197c4cc7ce9e88f11: No bootstrap required, opened a new log
I20260927 02:41:15.469837 13721 master_runner.cc:186] Error getting master registration for 127.0.98.61:41151: OK, instance_id { permanent_uuid: "d3f339de7597470197c4cc7ce9e88f11" instance_seqno: 1790476875195011 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 02:41:15.472754 13721 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 02:41:15.473757 13864 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3f339de7597470197c4cc7ce9e88f11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51aec98a76574a3cadaf615315bbbb0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35145 } } peers { permanent_uuid: "d3f339de7597470197c4cc7ce9e88f11" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41151 } } peers { permanent_uuid: "42ee57ed2af04a7687f89fe8549d1dc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40715 } }
I20260927 02:41:15.473925 13864 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3f339de7597470197c4cc7ce9e88f11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:41:15.473973 13864 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3f339de7597470197c4cc7ce9e88f11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3f339de7597470197c4cc7ce9e88f11, State: Initialized, Role: FOLLOWER
I20260927 02:41:15.474421 13864 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3f339de7597470197c4cc7ce9e88f11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51aec98a76574a3cadaf615315bbbb0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35145 } } peers { permanent_uuid: "d3f339de7597470197c4cc7ce9e88f11" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41151 } } peers { permanent_uuid: "42ee57ed2af04a7687f89fe8549d1dc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40715 } }
W20260927 02:41:15.474551 13742 tablet.cc:2369] T 00000000000000000000000000000000 P 51aec98a76574a3cadaf615315bbbb0b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:41:15.474828 13982 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3f339de7597470197c4cc7ce9e88f11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51aec98a76574a3cadaf615315bbbb0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35145 } } peers { permanent_uuid: "d3f339de7597470197c4cc7ce9e88f11" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41151 } } peers { permanent_uuid: "42ee57ed2af04a7687f89fe8549d1dc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40715 } } }
I20260927 02:41:15.475008 13982 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3f339de7597470197c4cc7ce9e88f11 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:41:15.475684 13864 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3f339de7597470197c4cc7ce9e88f11 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 02:41:15.480685 13993 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d3f339de7597470197c4cc7ce9e88f11: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:41:15.480808 13993 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260927 02:41:15.481379 13981 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 51aec98a76574a3cadaf615315bbbb0b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:41:15.481501 13981 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:41:15.484812 13796 master_runner.cc:263] No leader master found from master d3f339de7597470197c4cc7ce9e88f11
I20260927 02:41:15.484879 13796 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 02:41:15.566296 13955 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42ee57ed2af04a7687f89fe8549d1dc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:41:15.566470 13955 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42ee57ed2af04a7687f89fe8549d1dc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51aec98a76574a3cadaf615315bbbb0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35145 } } peers { permanent_uuid: "d3f339de7597470197c4cc7ce9e88f11" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41151 } } peers { permanent_uuid: "42ee57ed2af04a7687f89fe8549d1dc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40715 } }
I20260927 02:41:15.567340 13955 leader_election.cc:288] T 00000000000000000000000000000000 P 42ee57ed2af04a7687f89fe8549d1dc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51aec98a76574a3cadaf615315bbbb0b (127.0.98.62:35145), d3f339de7597470197c4cc7ce9e88f11 (127.0.98.61:41151)
I20260927 02:41:15.567837 13763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "42ee57ed2af04a7687f89fe8549d1dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51aec98a76574a3cadaf615315bbbb0b" is_pre_election: true
I20260927 02:41:15.567948 13836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "42ee57ed2af04a7687f89fe8549d1dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3f339de7597470197c4cc7ce9e88f11" is_pre_election: true
I20260927 02:41:15.568143 13763 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 51aec98a76574a3cadaf615315bbbb0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42ee57ed2af04a7687f89fe8549d1dc4 in term 0.
I20260927 02:41:15.568257 13836 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d3f339de7597470197c4cc7ce9e88f11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42ee57ed2af04a7687f89fe8549d1dc4 in term 0.
I20260927 02:41:15.568683 13889 leader_election.cc:302] T 00000000000000000000000000000000 P 42ee57ed2af04a7687f89fe8549d1dc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42ee57ed2af04a7687f89fe8549d1dc4, d3f339de7597470197c4cc7ce9e88f11; no voters:
I20260927 02:41:15.568975 13955 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42ee57ed2af04a7687f89fe8549d1dc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:41:15.569051 13955 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42ee57ed2af04a7687f89fe8549d1dc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:41:15.569089 13955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42ee57ed2af04a7687f89fe8549d1dc4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:41:15.570395 13955 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42ee57ed2af04a7687f89fe8549d1dc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51aec98a76574a3cadaf615315bbbb0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35145 } } peers { permanent_uuid: "d3f339de7597470197c4cc7ce9e88f11" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41151 } } peers { permanent_uuid: "42ee57ed2af04a7687f89fe8549d1dc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40715 } }
I20260927 02:41:15.571843 13955 leader_election.cc:288] T 00000000000000000000000000000000 P 42ee57ed2af04a7687f89fe8549d1dc4 [CANDIDATE]: Term 1 election: Requested vote from peers 51aec98a76574a3cadaf615315bbbb0b (127.0.98.62:35145), d3f339de7597470197c4cc7ce9e88f11 (127.0.98.61:41151)
I20260927 02:41:15.572685 13763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "42ee57ed2af04a7687f89fe8549d1dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51aec98a76574a3cadaf615315bbbb0b"
I20260927 02:41:15.572806 13763 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51aec98a76574a3cadaf615315bbbb0b [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:41:15.574100 13763 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 51aec98a76574a3cadaf615315bbbb0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42ee57ed2af04a7687f89fe8549d1dc4 in term 1.
I20260927 02:41:15.574996 13888 leader_election.cc:302] T 00000000000000000000000000000000 P 42ee57ed2af04a7687f89fe8549d1dc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42ee57ed2af04a7687f89fe8549d1dc4, 51aec98a76574a3cadaf615315bbbb0b; no voters:
I20260927 02:41:15.575861 13836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "42ee57ed2af04a7687f89fe8549d1dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3f339de7597470197c4cc7ce9e88f11"
I20260927 02:41:15.575984 13836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3f339de7597470197c4cc7ce9e88f11 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:41:15.577325 13836 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d3f339de7597470197c4cc7ce9e88f11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42ee57ed2af04a7687f89fe8549d1dc4 in term 1.
I20260927 02:41:15.578097 13955 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42ee57ed2af04a7687f89fe8549d1dc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:41:15.578388 13955 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42ee57ed2af04a7687f89fe8549d1dc4 [term 1 LEADER]: Becoming Leader. State: Replica: 42ee57ed2af04a7687f89fe8549d1dc4, State: Running, Role: LEADER
I20260927 02:41:15.578737 13955 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42ee57ed2af04a7687f89fe8549d1dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51aec98a76574a3cadaf615315bbbb0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35145 } } peers { permanent_uuid: "d3f339de7597470197c4cc7ce9e88f11" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41151 } } peers { permanent_uuid: "42ee57ed2af04a7687f89fe8549d1dc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40715 } }
I20260927 02:41:15.580513 13955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42ee57ed2af04a7687f89fe8549d1dc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42ee57ed2af04a7687f89fe8549d1dc4. Latest consensus state: current_term: 1 leader_uuid: "42ee57ed2af04a7687f89fe8549d1dc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51aec98a76574a3cadaf615315bbbb0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35145 } } peers { permanent_uuid: "d3f339de7597470197c4cc7ce9e88f11" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41151 } } peers { permanent_uuid: "42ee57ed2af04a7687f89fe8549d1dc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40715 } } }
I20260927 02:41:15.580691 13955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42ee57ed2af04a7687f89fe8549d1dc4 [sys.catalog]: This master's current role is: LEADER
I20260927 02:41:15.582397 13998 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:41:15.583357 13998 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:41:15.588583 13836 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d3f339de7597470197c4cc7ce9e88f11 [term 1 FOLLOWER]: Refusing update from remote peer 42ee57ed2af04a7687f89fe8549d1dc4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:41:15.589287 13996 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 42ee57ed2af04a7687f89fe8549d1dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3f339de7597470197c4cc7ce9e88f11" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:41:15.593299 13982 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3f339de7597470197c4cc7ce9e88f11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42ee57ed2af04a7687f89fe8549d1dc4. Latest consensus state: current_term: 1 leader_uuid: "42ee57ed2af04a7687f89fe8549d1dc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51aec98a76574a3cadaf615315bbbb0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35145 } } peers { permanent_uuid: "d3f339de7597470197c4cc7ce9e88f11" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41151 } } peers { permanent_uuid: "42ee57ed2af04a7687f89fe8549d1dc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40715 } } }
I20260927 02:41:15.593410 13982 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3f339de7597470197c4cc7ce9e88f11 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:41:15.596755 13763 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 51aec98a76574a3cadaf615315bbbb0b [term 1 FOLLOWER]: Refusing update from remote peer 42ee57ed2af04a7687f89fe8549d1dc4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:41:15.597345 13955 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 42ee57ed2af04a7687f89fe8549d1dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51aec98a76574a3cadaf615315bbbb0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:41:15.601425 13969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51aec98a76574a3cadaf615315bbbb0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42ee57ed2af04a7687f89fe8549d1dc4. Latest consensus state: current_term: 1 leader_uuid: "42ee57ed2af04a7687f89fe8549d1dc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51aec98a76574a3cadaf615315bbbb0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35145 } } peers { permanent_uuid: "d3f339de7597470197c4cc7ce9e88f11" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41151 } } peers { permanent_uuid: "42ee57ed2af04a7687f89fe8549d1dc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40715 } } }
I20260927 02:41:15.601553 13969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51aec98a76574a3cadaf615315bbbb0b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:41:15.606204 13996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42ee57ed2af04a7687f89fe8549d1dc4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "42ee57ed2af04a7687f89fe8549d1dc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51aec98a76574a3cadaf615315bbbb0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35145 } } peers { permanent_uuid: "d3f339de7597470197c4cc7ce9e88f11" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41151 } } peers { permanent_uuid: "42ee57ed2af04a7687f89fe8549d1dc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40715 } } }
I20260927 02:41:15.606345 13996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42ee57ed2af04a7687f89fe8549d1dc4 [sys.catalog]: This master's current role is: LEADER
I20260927 02:41:15.607573 13969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51aec98a76574a3cadaf615315bbbb0b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "42ee57ed2af04a7687f89fe8549d1dc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51aec98a76574a3cadaf615315bbbb0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35145 } } peers { permanent_uuid: "d3f339de7597470197c4cc7ce9e88f11" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41151 } } peers { permanent_uuid: "42ee57ed2af04a7687f89fe8549d1dc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40715 } } }
I20260927 02:41:15.607683 13969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51aec98a76574a3cadaf615315bbbb0b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:41:15.608119 13999 mvcc.cc:205] Tried to move back new op lower bound from 7333793282403930112 to 7333793282374098944. Current Snapshot: MvccSnapshot[applied={T|T < 7333793282403930112}]
I20260927 02:41:15.610028 14002 mvcc.cc:205] Tried to move back new op lower bound from 7333793282403930112 to 7333793282374098944. Current Snapshot: MvccSnapshot[applied={T|T < 7333793282403930112}]
I20260927 02:41:15.610831 13955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42ee57ed2af04a7687f89fe8549d1dc4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "42ee57ed2af04a7687f89fe8549d1dc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51aec98a76574a3cadaf615315bbbb0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35145 } } peers { permanent_uuid: "d3f339de7597470197c4cc7ce9e88f11" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41151 } } peers { permanent_uuid: "42ee57ed2af04a7687f89fe8549d1dc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40715 } } }
I20260927 02:41:15.610939 13955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42ee57ed2af04a7687f89fe8549d1dc4 [sys.catalog]: This master's current role is: LEADER
I20260927 02:41:15.613844 13982 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3f339de7597470197c4cc7ce9e88f11 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "42ee57ed2af04a7687f89fe8549d1dc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51aec98a76574a3cadaf615315bbbb0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35145 } } peers { permanent_uuid: "d3f339de7597470197c4cc7ce9e88f11" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41151 } } peers { permanent_uuid: "42ee57ed2af04a7687f89fe8549d1dc4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40715 } } }
I20260927 02:41:15.613953 13982 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3f339de7597470197c4cc7ce9e88f11 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:41:15.615257 14000 mvcc.cc:205] Tried to move back new op lower bound from 7333793282403930112 to 7333793282374098944. Current Snapshot: MvccSnapshot[applied={T|T < 7333793282403930112}]
I20260927 02:41:15.618868 13998 catalog_manager.cc:1347] Generated new cluster ID: 73c63ec3c5544e8d854f99e1ddbd1f64
I20260927 02:41:15.618961 13998 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:41:15.665469 13998 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:41:15.667073 13998 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:41:15.683581 13998 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42ee57ed2af04a7687f89fe8549d1dc4: Generated new TSK 0
I20260927 02:41:15.684480 13998 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:41:16.482177 13993 catalog_manager.cc:1259] Loaded cluster ID: 73c63ec3c5544e8d854f99e1ddbd1f64
I20260927 02:41:16.482290 13993 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d3f339de7597470197c4cc7ce9e88f11: loading cluster ID for follower catalog manager: success
I20260927 02:41:16.482671 13981 catalog_manager.cc:1259] Loaded cluster ID: 73c63ec3c5544e8d854f99e1ddbd1f64
I20260927 02:41:16.482728 13981 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 51aec98a76574a3cadaf615315bbbb0b: loading cluster ID for follower catalog manager: success
I20260927 02:41:16.483940 13993 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d3f339de7597470197c4cc7ce9e88f11: acquiring CA information for follower catalog manager: success
I20260927 02:41:16.484292 13981 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 51aec98a76574a3cadaf615315bbbb0b: acquiring CA information for follower catalog manager: success
I20260927 02:41:16.485044 13993 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d3f339de7597470197c4cc7ce9e88f11: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 02:41:16.485270 13981 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 51aec98a76574a3cadaf615315bbbb0b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 02:41:16.756289 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 13721
I20260927 02:41:16.761497 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 13796
I20260927 02:41:16.766217 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 13871
2026-09-27T02:41:16Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3041 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476718023131-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:41:16 dist-test-slave-zv8z krb5kdc[14014](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:41:16 dist-test-slave-zv8z krb5kdc[14014](info): set up 2 sockets
Sep 27 02:41:16 dist-test-slave-zv8z krb5kdc[14014](info): commencing operation
krb5kdc: starting...
W20260927 02:41:17.718200 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476718023131-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476718023131-392-0/krb5kdc/test-user.keytab.
Sep 27 02:41:17 dist-test-slave-zv8z krb5kdc[14014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:41:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:41:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:41:17.833323 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45803
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45803
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:41:17.942337 14027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:41:17.942631 14027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:41:17.942672 14027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:41:17.946764 14027 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:41:17.946836 14027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:41:17.946856 14027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:41:17.946870 14027 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:41:17.946883 14027 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:41:17.952275 14027 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:41:17.953325 14027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:41:17.954367 14027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:41:17.962350 14032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:41:17.962366 14033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:41:17.962476 14027 server_base.cc:1034] running on GCE node
W20260927 02:41:17.962715 14035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:41:17.963238 14027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:41:17.964280 14027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:41:17.965441 14027 hybrid_clock.cc:648] HybridClock initialized: now 1790476877965415 us; error 38 us; skew 500 ppm
Sep 27 02:41:17 dist-test-slave-zv8z krb5kdc[14014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:41:17.967975 14027 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:41:17.968819 14027 webserver.cc:460] Webserver started at http://127.0.98.62:39215/ using document root <none> and password file <none>
I20260927 02:41:17.969328 14027 fs_manager.cc:359] Metadata directory not provided
I20260927 02:41:17.969391 14027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:41:17.969574 14027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:41:17.971556 14027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476718023131-392-0/minicluster-data/master-0/data/instance:
uuid: "394b9976a8e14d4b8b1abd9501967116"
format_stamp: "Formatted at 2026-09-27 02:41:17 on dist-test-slave-zv8z"
server_key: "aae8a3fc5b04114c490cc45d5fbc32ad"
server_key_iv: "e457995bc875d4918206b4fee5b30d38"
server_key_version: "encryptionkey@0"
I20260927 02:41:17.972069 14027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476718023131-392-0/minicluster-data/master-0/wal/instance:
uuid: "394b9976a8e14d4b8b1abd9501967116"
format_stamp: "Formatted at 2026-09-27 02:41:17 on dist-test-slave-zv8z"
server_key: "aae8a3fc5b04114c490cc45d5fbc32ad"
server_key_iv: "e457995bc875d4918206b4fee5b30d38"
server_key_version: "encryptionkey@0"
I20260927 02:41:17.975471 14027 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:41:17.977669 14042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:17.978842 14027 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:41:17.978946 14027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "394b9976a8e14d4b8b1abd9501967116"
format_stamp: "Formatted at 2026-09-27 02:41:17 on dist-test-slave-zv8z"
server_key: "aae8a3fc5b04114c490cc45d5fbc32ad"
server_key_iv: "e457995bc875d4918206b4fee5b30d38"
server_key_version: "encryptionkey@0"
I20260927 02:41:17.979036 14027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:41:18.011653 14027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:41:18.014622 14027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:41:18.014796 14027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:41:18.021796 14027 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45803
I20260927 02:41:18.021816 14094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45803 every 8 connection(s)
I20260927 02:41:18.022843 14027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:41:18.025740 14095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:41:18.029501 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 14027
I20260927 02:41:18.029623 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790476718023131-392-0/minicluster-data/master-0/wal/instance
I20260927 02:41:18.029928 392 external_mini_cluster.cc:1428] Setting key 80c289d6712e3b666326ee7775961887
I20260927 02:41:18.031836 14095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 394b9976a8e14d4b8b1abd9501967116: Bootstrap starting.
I20260927 02:41:18.034011 14095 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 394b9976a8e14d4b8b1abd9501967116: Neither blocks nor log segments found. Creating new log.
I20260927 02:41:18.034773 14095 log.cc:824] T 00000000000000000000000000000000 P 394b9976a8e14d4b8b1abd9501967116: Log is configured to *not* fsync() on all Append() calls
I20260927 02:41:18.036875 14095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 394b9976a8e14d4b8b1abd9501967116: No bootstrap required, opened a new log
Sep 27 02:41:18 dist-test-slave-zv8z krb5kdc[14014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:41:18.041121 14095 raft_consensus.cc:348] T 00000000000000000000000000000000 P 394b9976a8e14d4b8b1abd9501967116 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "394b9976a8e14d4b8b1abd9501967116" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45803 } }
I20260927 02:41:18.041385 14095 raft_consensus.cc:374] T 00000000000000000000000000000000 P 394b9976a8e14d4b8b1abd9501967116 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:41:18.041441 14095 raft_consensus.cc:729] T 00000000000000000000000000000000 P 394b9976a8e14d4b8b1abd9501967116 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 394b9976a8e14d4b8b1abd9501967116, State: Initialized, Role: FOLLOWER
I20260927 02:41:18.041895 14095 consensus_queue.cc:260] T 00000000000000000000000000000000 P 394b9976a8e14d4b8b1abd9501967116 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "394b9976a8e14d4b8b1abd9501967116" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45803 } }
I20260927 02:41:18.042038 14095 raft_consensus.cc:388] T 00000000000000000000000000000000 P 394b9976a8e14d4b8b1abd9501967116 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:41:18.042105 14095 raft_consensus.cc:482] T 00000000000000000000000000000000 P 394b9976a8e14d4b8b1abd9501967116 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:41:18.042196 14095 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 394b9976a8e14d4b8b1abd9501967116 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:41:18.043135 14095 raft_consensus.cc:504] T 00000000000000000000000000000000 P 394b9976a8e14d4b8b1abd9501967116 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "394b9976a8e14d4b8b1abd9501967116" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45803 } }
I20260927 02:41:18.043507 14095 leader_election.cc:302] T 00000000000000000000000000000000 P 394b9976a8e14d4b8b1abd9501967116 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 394b9976a8e14d4b8b1abd9501967116; no voters:
I20260927 02:41:18.043843 14095 leader_election.cc:288] T 00000000000000000000000000000000 P 394b9976a8e14d4b8b1abd9501967116 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:41:18.043960 14100 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 394b9976a8e14d4b8b1abd9501967116 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:41:18.044185 14100 raft_consensus.cc:686] T 00000000000000000000000000000000 P 394b9976a8e14d4b8b1abd9501967116 [term 1 LEADER]: Becoming Leader. State: Replica: 394b9976a8e14d4b8b1abd9501967116, State: Running, Role: LEADER
I20260927 02:41:18.044555 14100 consensus_queue.cc:237] T 00000000000000000000000000000000 P 394b9976a8e14d4b8b1abd9501967116 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "394b9976a8e14d4b8b1abd9501967116" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45803 } }
I20260927 02:41:18.044776 14098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:18.031336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53952 (local address 127.0.98.62:45803)
0927 02:41:18.031768 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:18.031784 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:18.031820 (+ 36us) server_negotiation.cc:407] Connection header received
0927 02:41:18.032477 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:18.032495 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:18.032899 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:18.033210 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:41:18.034145 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:18.035310 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:18.036520 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:18.036996 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:18.037136 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:18.039716 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:41:18.039723 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:41:18.039741 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:41:18.039776 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:41:18.041634 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:18.042088 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:18.042094 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:18.042099 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:18.042238 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:18.042622 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:18.042625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:18.042627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:18.043003 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:41:18.043157 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:18.043457 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 02:41:18.043742 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":134,"threads_started":1}
I20260927 02:41:18.045027 14095 sys_catalog.cc:564] T 00000000000000000000000000000000 P 394b9976a8e14d4b8b1abd9501967116 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:41:18.046060 14102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 394b9976a8e14d4b8b1abd9501967116 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "394b9976a8e14d4b8b1abd9501967116" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "394b9976a8e14d4b8b1abd9501967116" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45803 } } }
I20260927 02:41:18.046172 14102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 394b9976a8e14d4b8b1abd9501967116 [sys.catalog]: This master's current role is: LEADER
I20260927 02:41:18.046409 14102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 394b9976a8e14d4b8b1abd9501967116 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 394b9976a8e14d4b8b1abd9501967116. Latest consensus state: current_term: 1 leader_uuid: "394b9976a8e14d4b8b1abd9501967116" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "394b9976a8e14d4b8b1abd9501967116" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45803 } } }
I20260927 02:41:18.046478 14102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 394b9976a8e14d4b8b1abd9501967116 [sys.catalog]: This master's current role is: LEADER
I20260927 02:41:18.046855 14109 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:41:18.047829 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:41:18.050165 14109 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:41:18.055943 14109 catalog_manager.cc:1347] Generated new cluster ID: 53b071b829994c2d9aca83edfea1da0e
I20260927 02:41:18.056007 14109 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:41:18.070999 14109 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:41:18.071985 14109 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:41:18.079821 14109 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 394b9976a8e14d4b8b1abd9501967116: Generated new TSK 0
I20260927 02:41:18.080543 14109 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:41:18.103243 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 14027
2026-09-27T02:41:18Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1336 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:41:18 dist-test-slave-zv8z krb5kdc[14122](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:41:18 dist-test-slave-zv8z krb5kdc[14122](info): set up 2 sockets
Sep 27 02:41:18 dist-test-slave-zv8z krb5kdc[14122](info): commencing operation
krb5kdc: starting...
W20260927 02:41:19.054229 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/test-user.keytab.
Sep 27 02:41:19 dist-test-slave-zv8z krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:41:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:41:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:41:19.169272 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45947
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45947
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:41:19.277233 14135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:41:19.277506 14135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:41:19.277544 14135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:41:19.281510 14135 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:41:19.281572 14135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:41:19.281591 14135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:41:19.281607 14135 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:41:19.281621 14135 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:41:19.286897 14135 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:41:19.287923 14135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:41:19.288916 14135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:41:19.297088 14141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:41:19.297142 14143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:41:19.297096 14140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:41:19.297164 14135 server_base.cc:1034] running on GCE node
I20260927 02:41:19.297844 14135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:41:19.298909 14135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:41:19.300105 14135 hybrid_clock.cc:648] HybridClock initialized: now 1790476879300069 us; error 42 us; skew 500 ppm
Sep 27 02:41:19 dist-test-slave-zv8z krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476879, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:41:19.308332 14135 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:41:19.309404 14135 webserver.cc:460] Webserver started at http://127.0.98.62:32953/ using document root <none> and password file <none>
I20260927 02:41:19.309993 14135 fs_manager.cc:359] Metadata directory not provided
I20260927 02:41:19.310067 14135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:41:19.310331 14135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:41:19.312426 14135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/master-0/data/instance:
uuid: "d7409a2403734592bcce742551f20997"
format_stamp: "Formatted at 2026-09-27 02:41:19 on dist-test-slave-zv8z"
server_key: "32528d19bb181df7aa255d8c2a4e30f8"
server_key_iv: "b0beda8656ece766f07094794636c357"
server_key_version: "encryptionkey@0"
I20260927 02:41:19.312964 14135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/master-0/wal/instance:
uuid: "d7409a2403734592bcce742551f20997"
format_stamp: "Formatted at 2026-09-27 02:41:19 on dist-test-slave-zv8z"
server_key: "32528d19bb181df7aa255d8c2a4e30f8"
server_key_iv: "b0beda8656ece766f07094794636c357"
server_key_version: "encryptionkey@0"
I20260927 02:41:19.316787 14135 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 02:41:19.319316 14150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:19.320649 14135 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:41:19.320804 14135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "d7409a2403734592bcce742551f20997"
format_stamp: "Formatted at 2026-09-27 02:41:19 on dist-test-slave-zv8z"
server_key: "32528d19bb181df7aa255d8c2a4e30f8"
server_key_iv: "b0beda8656ece766f07094794636c357"
server_key_version: "encryptionkey@0"
I20260927 02:41:19.320905 14135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:41:19.348062 14135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:41:19.351450 14135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:41:19.351745 14135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:41:19.358731 14135 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45947
I20260927 02:41:19.359011 14202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45947 every 8 connection(s)
I20260927 02:41:19.359684 14135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:41:19.362571 14203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:41:19.365331 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 14135
I20260927 02:41:19.365449 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/master-0/wal/instance
I20260927 02:41:19.365697 392 external_mini_cluster.cc:1428] Setting key 1878a733913237dd800f77a600641ad2
I20260927 02:41:19.368595 14203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997: Bootstrap starting.
I20260927 02:41:19.371284 14203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997: Neither blocks nor log segments found. Creating new log.
I20260927 02:41:19.372038 14203 log.cc:824] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997: Log is configured to *not* fsync() on all Append() calls
I20260927 02:41:19.374068 14203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997: No bootstrap required, opened a new log
Sep 27 02:41:19 dist-test-slave-zv8z krb5kdc[14122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:41:19.377810 14203 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7409a2403734592bcce742551f20997" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45947 } }
I20260927 02:41:19.378087 14203 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:41:19.378155 14203 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7409a2403734592bcce742551f20997, State: Initialized, Role: FOLLOWER
I20260927 02:41:19.378626 14203 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7409a2403734592bcce742551f20997" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45947 } }
I20260927 02:41:19.378785 14203 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:41:19.378852 14203 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:41:19.378949 14203 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:41:19.380028 14203 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7409a2403734592bcce742551f20997" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45947 } }
I20260927 02:41:19.380410 14203 leader_election.cc:302] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7409a2403734592bcce742551f20997; no voters:
I20260927 02:41:19.380771 14203 leader_election.cc:288] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:41:19.380874 14208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:41:19.381089 14208 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997 [term 1 LEADER]: Becoming Leader. State: Replica: d7409a2403734592bcce742551f20997, State: Running, Role: LEADER
I20260927 02:41:19.381350 14206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:19.367082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50662 (local address 127.0.98.62:45947)
0927 02:41:19.367521 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:19.367541 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:19.367597 (+ 56us) server_negotiation.cc:407] Connection header received
0927 02:41:19.368368 (+ 771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:19.368390 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:19.368833 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:19.369197 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:41:19.369924 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:19.371419 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:19.372291 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:19.372969 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:19.373120 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:19.375771 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:41:19.375777 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:41:19.375793 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:41:19.375825 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:41:19.378272 (+ 2447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:19.378694 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:19.378700 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:19.378706 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:19.378845 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:19.379097 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:19.379100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:19.379102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:19.379596 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:41:19.379799 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:19.380086 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 02:41:19.380386 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":127,"threads_started":1}
I20260927 02:41:19.381475 14208 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7409a2403734592bcce742551f20997" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45947 } }
I20260927 02:41:19.382081 14203 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:41:19.383208 14210 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7409a2403734592bcce742551f20997. Latest consensus state: current_term: 1 leader_uuid: "d7409a2403734592bcce742551f20997" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7409a2403734592bcce742551f20997" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45947 } } }
I20260927 02:41:19.383352 14210 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997 [sys.catalog]: This master's current role is: LEADER
I20260927 02:41:19.383212 14209 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7409a2403734592bcce742551f20997" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7409a2403734592bcce742551f20997" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45947 } } }
I20260927 02:41:19.383433 14209 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997 [sys.catalog]: This master's current role is: LEADER
I20260927 02:41:19.384482 14212 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:41:19.387341 14212 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:41:19.392686 14212 catalog_manager.cc:1347] Generated new cluster ID: 113bd70eda5b40158c169ffd208d3f94
I20260927 02:41:19.392762 14212 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:41:19.414600 14212 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:41:19.415889 14212 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:41:19.422971 14212 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997: Generated new TSK 0
I20260927 02:41:19.423846 14212 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:41:19.453507 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45947
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:41:19.564198 14229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:41:19.564468 14229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:41:19.564509 14229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:41:19.568523 14229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:41:19.568581 14229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:41:19.568645 14229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:41:19.573927 14229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:41:19.574972 14229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:41:19.575973 14229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:41:19.585572 14234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:41:19.585557 14237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:41:19.585632 14235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:41:19.585793 14229 server_base.cc:1034] running on GCE node
I20260927 02:41:19.586323 14229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:41:19.587025 14229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:41:19.588181 14229 hybrid_clock.cc:648] HybridClock initialized: now 1790476879588148 us; error 52 us; skew 500 ppm
Sep 27 02:41:19 dist-test-slave-zv8z krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476879, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:41:19.590919 14229 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:41:19.591779 14229 webserver.cc:460] Webserver started at http://127.0.98.1:41129/ using document root <none> and password file <none>
I20260927 02:41:19.592324 14229 fs_manager.cc:359] Metadata directory not provided
I20260927 02:41:19.592391 14229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:41:19.592597 14229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:41:19.594647 14229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-0/data/instance:
uuid: "37e1e5c5ca634982a0a258e89333bf26"
format_stamp: "Formatted at 2026-09-27 02:41:19 on dist-test-slave-zv8z"
server_key: "f6671aefcba165e019246ed6a0575046"
server_key_iv: "80b76de96e47d39b5ce1ae98f3f8539f"
server_key_version: "encryptionkey@0"
I20260927 02:41:19.595178 14229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-0/wal/instance:
uuid: "37e1e5c5ca634982a0a258e89333bf26"
format_stamp: "Formatted at 2026-09-27 02:41:19 on dist-test-slave-zv8z"
server_key: "f6671aefcba165e019246ed6a0575046"
server_key_iv: "80b76de96e47d39b5ce1ae98f3f8539f"
server_key_version: "encryptionkey@0"
I20260927 02:41:19.598698 14229 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:41:19.600929 14244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:19.602098 14229 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:41:19.602200 14229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-0/wal
uuid: "37e1e5c5ca634982a0a258e89333bf26"
format_stamp: "Formatted at 2026-09-27 02:41:19 on dist-test-slave-zv8z"
server_key: "f6671aefcba165e019246ed6a0575046"
server_key_iv: "80b76de96e47d39b5ce1ae98f3f8539f"
server_key_version: "encryptionkey@0"
I20260927 02:41:19.602342 14229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:41:19.628353 14229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:41:19.631712 14229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:41:19.631884 14229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:41:19.632467 14229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:41:19.633297 14229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:41:19.633365 14229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:19.633466 14229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:41:19.633496 14229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:19.641901 14229 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42077
I20260927 02:41:19.641948 14357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42077 every 8 connection(s)
I20260927 02:41:19.642841 14229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:41:19.649746 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 14229
I20260927 02:41:19.649878 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:41:19.650184 392 external_mini_cluster.cc:1428] Setting key dc4d30c5e18b4fca330e44fc8a7d7a6c
Sep 27 02:41:19 dist-test-slave-zv8z krb5kdc[14122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476879, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:41:19.656924 14206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:19.644760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60399 (local address 127.0.98.62:45947)
0927 02:41:19.644901 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:19.644906 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:19.645736 (+ 830us) server_negotiation.cc:407] Connection header received
0927 02:41:19.647003 (+ 1267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:19.647006 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:19.647051 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:19.647125 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:41:19.648259 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:19.648800 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:19.650103 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:19.650386 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:19.650464 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:19.652993 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:41:19.652998 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:41:19.653000 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:41:19.653029 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:41:19.654504 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:19.655044 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:19.655048 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:19.655050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:19.655161 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:19.655518 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:19.655522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:19.655524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:19.655798 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:41:19.655896 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:19.656616 (+ 720us) server_negotiation.cc:299] Negotiation successful
0927 02:41:19.656756 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 02:41:19.658493 14360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:19.645141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.1:60399)
0927 02:41:19.645589 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:19.645632 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 02:41:19.646810 (+ 1178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:41:19.647278 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:41:19.647286 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:41:19.647622 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:41:19.648127 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:19.648141 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:19.648907 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:19.648910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:19.649985 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:19.649996 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:19.650527 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:19.650531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:19.650692 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:19.651230 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:41:19.651249 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:41:19.652825 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:41:19.654608 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:19.654614 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:19.654629 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:19.654935 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:19.655254 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:19.655258 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:19.655274 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:19.655438 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:19.655952 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:41:19.655958 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:41:19.656320 (+ 362us) client_negotiation.cc:769] Sending connection context
0927 02:41:19.657134 (+ 814us) client_negotiation.cc:240] Negotiation successful
0927 02:41:19.657479 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:41:19.662782 14358 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45947
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 02:41:19.663168 14358 heartbeater.cc:464] Registering TS with master...
I20260927 02:41:19.663879 14358 heartbeater.cc:511] Master 127.0.98.62:45947 requested a full tablet report, sending...
I20260927 02:41:19.665723 14167 ts_manager.cc:194] Registered new tserver with Master: 37e1e5c5ca634982a0a258e89333bf26 (127.0.98.1:42077)
I20260927 02:41:19.667294 14167 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60399
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:41:19.678895 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45947
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:41:19.784610 14363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:41:19.784919 14363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:41:19.784987 14363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:41:19.789031 14363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:41:19.789109 14363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:41:19.789203 14363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:41:19.794474 14363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:41:19.795490 14363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:41:19.796476 14363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:41:19.805702 14371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:41:19.805675 14369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:41:19.805682 14368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:41:19.805967 14363 server_base.cc:1034] running on GCE node
I20260927 02:41:19.806427 14363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:41:19.807056 14363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:41:19.808224 14363 hybrid_clock.cc:648] HybridClock initialized: now 1790476879808196 us; error 38 us; skew 500 ppm
Sep 27 02:41:19 dist-test-slave-zv8z krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476879, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:41:19.810850 14363 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:41:19.811733 14363 webserver.cc:460] Webserver started at http://127.0.98.2:44993/ using document root <none> and password file <none>
I20260927 02:41:19.812250 14363 fs_manager.cc:359] Metadata directory not provided
I20260927 02:41:19.812314 14363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:41:19.812489 14363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:41:19.814458 14363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-1/data/instance:
uuid: "f52a7d693ec54ea19c27916605a799b4"
format_stamp: "Formatted at 2026-09-27 02:41:19 on dist-test-slave-zv8z"
server_key: "a0c4d48df42b09f807c9e078eb24d97a"
server_key_iv: "2e69127122017f700a318d37feb1df45"
server_key_version: "encryptionkey@0"
I20260927 02:41:19.814957 14363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f52a7d693ec54ea19c27916605a799b4"
format_stamp: "Formatted at 2026-09-27 02:41:19 on dist-test-slave-zv8z"
server_key: "a0c4d48df42b09f807c9e078eb24d97a"
server_key_iv: "2e69127122017f700a318d37feb1df45"
server_key_version: "encryptionkey@0"
I20260927 02:41:19.818470 14363 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:41:19.820808 14378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:19.822103 14363 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:41:19.822312 14363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-1/wal
uuid: "f52a7d693ec54ea19c27916605a799b4"
format_stamp: "Formatted at 2026-09-27 02:41:19 on dist-test-slave-zv8z"
server_key: "a0c4d48df42b09f807c9e078eb24d97a"
server_key_iv: "2e69127122017f700a318d37feb1df45"
server_key_version: "encryptionkey@0"
I20260927 02:41:19.822409 14363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:41:19.845201 14363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:41:19.848165 14363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:41:19.848315 14363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:41:19.848836 14363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:41:19.849612 14363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:41:19.849656 14363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:19.849694 14363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:41:19.849713 14363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:19.858230 14363 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36519
I20260927 02:41:19.858322 14491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36519 every 8 connection(s)
I20260927 02:41:19.859138 14363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:41:19.864472 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 14363
I20260927 02:41:19.864596 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:41:19.864877 392 external_mini_cluster.cc:1428] Setting key 8aeefea7de0123d22de3ca52c10ef350
Sep 27 02:41:19 dist-test-slave-zv8z krb5kdc[14122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476879, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:41:19.873601 14206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:19.860940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43581 (local address 127.0.98.62:45947)
0927 02:41:19.861082 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:19.861085 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:19.861925 (+ 840us) server_negotiation.cc:407] Connection header received
0927 02:41:19.863300 (+ 1375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:19.863303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:19.863357 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:19.863429 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:41:19.864764 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:19.865336 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:19.866807 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:19.867108 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:19.867227 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:19.869850 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:41:19.869854 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:41:19.869857 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:41:19.869885 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:41:19.871546 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:19.872052 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:19.872055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:19.872058 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:19.872149 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:19.872462 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:19.872466 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:19.872468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:19.872695 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:41:19.872791 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:19.873308 (+ 517us) server_negotiation.cc:299] Negotiation successful
0927 02:41:19.873432 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 02:41:19.874373 14494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:19.861312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.2:43581)
0927 02:41:19.861767 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:19.861812 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 02:41:19.863018 (+ 1206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:41:19.863618 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:41:19.863626 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:41:19.864006 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:41:19.864543 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:19.864565 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:19.865469 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:19.865472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:19.866679 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:19.866689 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:19.867258 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:19.867261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:19.867397 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:19.868030 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:41:19.868051 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:41:19.869673 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:41:19.871692 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:19.871697 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:19.871710 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:19.871960 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:19.872247 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:19.872249 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:19.872250 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:19.872384 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:19.872792 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:41:19.872798 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:41:19.873071 (+ 273us) client_negotiation.cc:769] Sending connection context
0927 02:41:19.873228 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 02:41:19.873482 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":99,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:41:19.875545 14492 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45947
I20260927 02:41:19.875794 14492 heartbeater.cc:464] Registering TS with master...
I20260927 02:41:19.876256 14492 heartbeater.cc:511] Master 127.0.98.62:45947 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 02:41:19.877321 14167 ts_manager.cc:194] Registered new tserver with Master: f52a7d693ec54ea19c27916605a799b4 (127.0.98.2:36519)
I20260927 02:41:19.877880 14167 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43581
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:41:19.892727 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45947
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:41:20.014566 14497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:41:20.014832 14497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:41:20.014875 14497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:41:20.018968 14497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:41:20.019026 14497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:41:20.019091 14497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:41:20.024418 14497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14497
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:41:20.025540 14497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:41:20.026647 14497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:41:20.036157 14503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:41:20.036216 14505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:41:20.036273 14497 server_base.cc:1034] running on GCE node
W20260927 02:41:20.036230 14502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:41:20.036789 14497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:41:20.037413 14497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:41:20.038594 14497 hybrid_clock.cc:648] HybridClock initialized: now 1790476880038572 us; error 32 us; skew 500 ppm
Sep 27 02:41:20 dist-test-slave-zv8z krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476880, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:41:20.041296 14497 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:41:20.042173 14497 webserver.cc:460] Webserver started at http://127.0.98.3:40909/ using document root <none> and password file <none>
I20260927 02:41:20.042726 14497 fs_manager.cc:359] Metadata directory not provided
I20260927 02:41:20.042785 14497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:41:20.042979 14497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:41:20.044950 14497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-2/data/instance:
uuid: "7baca47cd2b04cbf9a68e5f0f6305a6d"
format_stamp: "Formatted at 2026-09-27 02:41:20 on dist-test-slave-zv8z"
server_key: "bfdc1c2c2d5f6176ee1e7fc64974b164"
server_key_iv: "9f41d3d33bab1866beaa4909769508f3"
server_key_version: "encryptionkey@0"
I20260927 02:41:20.045457 14497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7baca47cd2b04cbf9a68e5f0f6305a6d"
format_stamp: "Formatted at 2026-09-27 02:41:20 on dist-test-slave-zv8z"
server_key: "bfdc1c2c2d5f6176ee1e7fc64974b164"
server_key_iv: "9f41d3d33bab1866beaa4909769508f3"
server_key_version: "encryptionkey@0"
I20260927 02:41:20.049052 14497 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 02:41:20.051455 14512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:20.052840 14497 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:41:20.052934 14497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-2/wal
uuid: "7baca47cd2b04cbf9a68e5f0f6305a6d"
format_stamp: "Formatted at 2026-09-27 02:41:20 on dist-test-slave-zv8z"
server_key: "bfdc1c2c2d5f6176ee1e7fc64974b164"
server_key_iv: "9f41d3d33bab1866beaa4909769508f3"
server_key_version: "encryptionkey@0"
I20260927 02:41:20.053048 14497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:41:20.068454 14497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:41:20.071875 14497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:41:20.072068 14497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:41:20.072664 14497 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:41:20.073501 14497 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:41:20.073545 14497 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:20.073589 14497 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:41:20.073618 14497 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:20.081854 14497 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44647
I20260927 02:41:20.081873 14625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44647 every 8 connection(s)
I20260927 02:41:20.082736 14497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:41:20.088882 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 14497
I20260927 02:41:20.089010 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:41:20.089310 392 external_mini_cluster.cc:1428] Setting key 95f6360607754b5cc43455ec635e9b4e
Sep 27 02:41:20 dist-test-slave-zv8z krb5kdc[14122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476880, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:41:20.096549 14206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:20.084548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56367 (local address 127.0.98.62:45947)
0927 02:41:20.084701 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:20.084705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:20.085541 (+ 836us) server_negotiation.cc:407] Connection header received
0927 02:41:20.086760 (+ 1219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:20.086763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:20.086808 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:20.086878 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:41:20.088029 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:20.088549 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:20.089973 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:20.090326 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:20.090421 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:20.092750 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:41:20.092754 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:41:20.092756 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:41:20.092782 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:41:20.094419 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:20.094984 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:20.094987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:20.094989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:20.095080 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:20.095474 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:20.095477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:20.095479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:20.095698 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:41:20.095784 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:20.096273 (+ 489us) server_negotiation.cc:299] Negotiation successful
0927 02:41:20.096389 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 02:41:20.097400 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:20.084957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.3:56367)
0927 02:41:20.085392 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:20.085433 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:41:20.086574 (+ 1141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:41:20.087023 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:41:20.087029 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:41:20.087417 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:41:20.087872 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:20.087885 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:20.088672 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:20.088676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:20.089867 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:20.089876 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:20.090465 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:20.090469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:20.090659 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:20.091208 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:41:20.091224 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:41:20.092601 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:41:20.094517 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:20.094523 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:20.094557 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:20.094894 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:20.095196 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:20.095198 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:20.095200 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:20.095374 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:20.095797 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:41:20.095804 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:41:20.096047 (+ 243us) client_negotiation.cc:769] Sending connection context
0927 02:41:20.096218 (+ 171us) client_negotiation.cc:240] Negotiation successful
0927 02:41:20.096538 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":112,"threads_started":1}
I20260927 02:41:20.098562 14626 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45947
I20260927 02:41:20.098816 14626 heartbeater.cc:464] Registering TS with master...
I20260927 02:41:20.099294 14626 heartbeater.cc:511] Master 127.0.98.62:45947 requested a full tablet report, sending...
I20260927 02:41:20.100310 14167 ts_manager.cc:194] Registered new tserver with Master: 7baca47cd2b04cbf9a68e5f0f6305a6d (127.0.98.3:44647)
I20260927 02:41:20.100867 14167 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56367
I20260927 02:41:20.102965 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:41:20.112157 14206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:20.104840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50670 (local address 127.0.98.62:45947)
0927 02:41:20.104977 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:20.104982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:20.105159 (+ 177us) server_negotiation.cc:407] Connection header received
0927 02:41:20.105264 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:20.105267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:20.105307 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:20.105372 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:41:20.106075 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:20.106757 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:20.107617 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:20.107844 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:20.107922 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:20.109271 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:41:20.109274 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:41:20.109276 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:41:20.109298 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:41:20.110612 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:20.111053 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:20.111056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:20.111057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:20.111146 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:20.111396 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:20.111399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:20.111400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:20.111622 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:41:20.111745 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:20.111912 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 02:41:20.112005 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 02:41:20.115003 14167 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50670:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:41:20.117997 14167 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:41:20.131815 14637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:20.126223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44647 (local address 127.0.0.1:41010)
0927 02:41:20.126604 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:20.126650 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 02:41:20.126775 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:41:20.127806 (+ 1031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:41:20.127812 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:41:20.127835 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:41:20.127961 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:20.127968 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:20.129037 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:20.129041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:20.130122 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:20.130132 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:20.131263 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:20.131267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:20.131385 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:20.131425 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 02:41:20.131537 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 02:41:20.131659 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":24,"thread_start_us":105,"threads_started":1}
I20260927 02:41:20.131830 14639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:20.126379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41010 (local address 127.0.98.3:44647)
0927 02:41:20.127137 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:20.127150 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:20.127174 (+ 24us) server_negotiation.cc:407] Connection header received
0927 02:41:20.127247 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:20.127252 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:20.127456 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:20.127605 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:41:20.128071 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:20.128920 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:20.130277 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:20.131150 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:20.131255 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:20.131392 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:20.131575 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 02:41:20.131686 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":159,"threads_started":1}
I20260927 02:41:20.132205 14638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:20.126370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42077 (local address 127.0.0.1:56312)
0927 02:41:20.127062 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:20.127076 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:41:20.127158 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:41:20.128151 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:41:20.128153 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:41:20.128172 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:41:20.128260 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:20.128266 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:20.129664 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:20.129668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:20.130811 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:20.130821 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:20.131847 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:20.131851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:20.131939 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:20.131962 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 02:41:20.132010 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 02:41:20.132070 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":119,"threads_started":1}
I20260927 02:41:20.132649 14641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:20.127091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36519 (local address 127.0.0.1:42650)
0927 02:41:20.127313 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:20.127324 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:41:20.127397 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:41:20.128245 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:41:20.128248 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:41:20.128266 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:41:20.128351 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:20.128357 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:20.129810 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:20.129813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:20.130908 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:20.130918 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:20.131915 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:20.131919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:20.132027 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:20.132052 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 02:41:20.132100 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 02:41:20.132167 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":3,"thread_start_us":73,"threads_started":1}
I20260927 02:41:20.132679 14642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:20.127456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42650 (local address 127.0.98.2:36519)
0927 02:41:20.127837 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:20.127847 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:20.127864 (+ 17us) server_negotiation.cc:407] Connection header received
0927 02:41:20.127942 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:20.127946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:20.128108 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:20.128484 (+ 376us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:41:20.128654 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:20.129699 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:20.131052 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:20.131783 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:20.132276 (+ 493us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:20.132414 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:20.132458 (+ 44us) server_negotiation.cc:299] Negotiation successful
0927 02:41:20.132551 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":117,"threads_started":1}
I20260927 02:41:20.133234 14640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:20.127019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56312 (local address 127.0.98.1:42077)
0927 02:41:20.127649 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:20.127663 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:20.127685 (+ 22us) server_negotiation.cc:407] Connection header received
0927 02:41:20.127759 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:20.127764 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:20.127964 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:20.128165 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:41:20.128348 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:20.129515 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:20.130952 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:20.131717 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:20.132880 (+ 1163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:20.133013 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:20.133051 (+ 38us) server_negotiation.cc:299] Negotiation successful
0927 02:41:20.133118 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":94,"threads_started":1}
I20260927 02:41:20.135252 14426 tablet_service.cc:1467] Processing CreateTablet for tablet 99ad8f8adaf24e06bf0ca8ee60abdd01 (DEFAULT_TABLE table=test-table [id=2b280bca6f2446aabfcf4053105305d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:41:20.135176 14560 tablet_service.cc:1467] Processing CreateTablet for tablet 99ad8f8adaf24e06bf0ca8ee60abdd01 (DEFAULT_TABLE table=test-table [id=2b280bca6f2446aabfcf4053105305d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:41:20.135699 14292 tablet_service.cc:1467] Processing CreateTablet for tablet 99ad8f8adaf24e06bf0ca8ee60abdd01 (DEFAULT_TABLE table=test-table [id=2b280bca6f2446aabfcf4053105305d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:41:20.136161 14426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99ad8f8adaf24e06bf0ca8ee60abdd01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:41:20.136211 14560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99ad8f8adaf24e06bf0ca8ee60abdd01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:41:20.136598 14292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99ad8f8adaf24e06bf0ca8ee60abdd01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:41:20.143246 14645 tablet_bootstrap.cc:492] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 37e1e5c5ca634982a0a258e89333bf26: Bootstrap starting.
I20260927 02:41:20.142084 14643 tablet_bootstrap.cc:492] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P f52a7d693ec54ea19c27916605a799b4: Bootstrap starting.
I20260927 02:41:20.145720 14645 tablet_bootstrap.cc:654] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 37e1e5c5ca634982a0a258e89333bf26: Neither blocks nor log segments found. Creating new log.
I20260927 02:41:20.146111 14643 tablet_bootstrap.cc:654] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P f52a7d693ec54ea19c27916605a799b4: Neither blocks nor log segments found. Creating new log.
I20260927 02:41:20.146307 14644 tablet_bootstrap.cc:492] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 7baca47cd2b04cbf9a68e5f0f6305a6d: Bootstrap starting.
I20260927 02:41:20.147050 14643 log.cc:824] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P f52a7d693ec54ea19c27916605a799b4: Log is configured to *not* fsync() on all Append() calls
I20260927 02:41:20.147051 14645 log.cc:824] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 37e1e5c5ca634982a0a258e89333bf26: Log is configured to *not* fsync() on all Append() calls
I20260927 02:41:20.148283 14644 tablet_bootstrap.cc:654] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 7baca47cd2b04cbf9a68e5f0f6305a6d: Neither blocks nor log segments found. Creating new log.
I20260927 02:41:20.149240 14643 tablet_bootstrap.cc:492] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P f52a7d693ec54ea19c27916605a799b4: No bootstrap required, opened a new log
I20260927 02:41:20.149240 14645 tablet_bootstrap.cc:492] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 37e1e5c5ca634982a0a258e89333bf26: No bootstrap required, opened a new log
I20260927 02:41:20.149443 14643 ts_tablet_manager.cc:1397] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P f52a7d693ec54ea19c27916605a799b4: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260927 02:41:20.149443 14645 ts_tablet_manager.cc:1397] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 37e1e5c5ca634982a0a258e89333bf26: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 02:41:20.149497 14644 log.cc:824] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 7baca47cd2b04cbf9a68e5f0f6305a6d: Log is configured to *not* fsync() on all Append() calls
I20260927 02:41:20.151232 14644 tablet_bootstrap.cc:492] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 7baca47cd2b04cbf9a68e5f0f6305a6d: No bootstrap required, opened a new log
I20260927 02:41:20.151408 14644 ts_tablet_manager.cc:1397] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 7baca47cd2b04cbf9a68e5f0f6305a6d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 02:41:20.153564 14643 raft_consensus.cc:348] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P f52a7d693ec54ea19c27916605a799b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7baca47cd2b04cbf9a68e5f0f6305a6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44647 } } peers { permanent_uuid: "37e1e5c5ca634982a0a258e89333bf26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42077 } } peers { permanent_uuid: "f52a7d693ec54ea19c27916605a799b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36519 } }
I20260927 02:41:20.153564 14645 raft_consensus.cc:348] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 37e1e5c5ca634982a0a258e89333bf26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7baca47cd2b04cbf9a68e5f0f6305a6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44647 } } peers { permanent_uuid: "37e1e5c5ca634982a0a258e89333bf26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42077 } } peers { permanent_uuid: "f52a7d693ec54ea19c27916605a799b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36519 } }
I20260927 02:41:20.153846 14643 raft_consensus.cc:374] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P f52a7d693ec54ea19c27916605a799b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:41:20.153846 14645 raft_consensus.cc:374] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 37e1e5c5ca634982a0a258e89333bf26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:41:20.153903 14643 raft_consensus.cc:729] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P f52a7d693ec54ea19c27916605a799b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f52a7d693ec54ea19c27916605a799b4, State: Initialized, Role: FOLLOWER
I20260927 02:41:20.153903 14645 raft_consensus.cc:729] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 37e1e5c5ca634982a0a258e89333bf26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37e1e5c5ca634982a0a258e89333bf26, State: Initialized, Role: FOLLOWER
I20260927 02:41:20.154204 14644 raft_consensus.cc:348] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 7baca47cd2b04cbf9a68e5f0f6305a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7baca47cd2b04cbf9a68e5f0f6305a6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44647 } } peers { permanent_uuid: "37e1e5c5ca634982a0a258e89333bf26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42077 } } peers { permanent_uuid: "f52a7d693ec54ea19c27916605a799b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36519 } }
I20260927 02:41:20.154464 14644 raft_consensus.cc:374] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 7baca47cd2b04cbf9a68e5f0f6305a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:41:20.154393 14643 consensus_queue.cc:260] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P f52a7d693ec54ea19c27916605a799b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7baca47cd2b04cbf9a68e5f0f6305a6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44647 } } peers { permanent_uuid: "37e1e5c5ca634982a0a258e89333bf26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42077 } } peers { permanent_uuid: "f52a7d693ec54ea19c27916605a799b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36519 } }
I20260927 02:41:20.154393 14645 consensus_queue.cc:260] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 37e1e5c5ca634982a0a258e89333bf26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7baca47cd2b04cbf9a68e5f0f6305a6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44647 } } peers { permanent_uuid: "37e1e5c5ca634982a0a258e89333bf26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42077 } } peers { permanent_uuid: "f52a7d693ec54ea19c27916605a799b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36519 } }
I20260927 02:41:20.154525 14644 raft_consensus.cc:729] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 7baca47cd2b04cbf9a68e5f0f6305a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7baca47cd2b04cbf9a68e5f0f6305a6d, State: Initialized, Role: FOLLOWER
I20260927 02:41:20.154942 14644 consensus_queue.cc:260] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 7baca47cd2b04cbf9a68e5f0f6305a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7baca47cd2b04cbf9a68e5f0f6305a6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44647 } } peers { permanent_uuid: "37e1e5c5ca634982a0a258e89333bf26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42077 } } peers { permanent_uuid: "f52a7d693ec54ea19c27916605a799b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36519 } }
I20260927 02:41:20.155424 14643 ts_tablet_manager.cc:1428] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P f52a7d693ec54ea19c27916605a799b4: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 02:41:20.155424 14645 ts_tablet_manager.cc:1428] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 37e1e5c5ca634982a0a258e89333bf26: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 02:41:20.155457 14358 heartbeater.cc:503] Master 127.0.98.62:45947 was elected leader, sending a full tablet report...
I20260927 02:41:20.155457 14492 heartbeater.cc:503] Master 127.0.98.62:45947 was elected leader, sending a full tablet report...
I20260927 02:41:20.156193 14644 ts_tablet_manager.cc:1428] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 7baca47cd2b04cbf9a68e5f0f6305a6d: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260927 02:41:20.156785 14626 heartbeater.cc:503] Master 127.0.98.62:45947 was elected leader, sending a full tablet report...
I20260927 02:41:20.288905 14650 raft_consensus.cc:482] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 37e1e5c5ca634982a0a258e89333bf26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:41:20.289126 14650 raft_consensus.cc:504] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 37e1e5c5ca634982a0a258e89333bf26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7baca47cd2b04cbf9a68e5f0f6305a6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44647 } } peers { permanent_uuid: "37e1e5c5ca634982a0a258e89333bf26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42077 } } peers { permanent_uuid: "f52a7d693ec54ea19c27916605a799b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36519 } }
I20260927 02:41:20.290274 14650 leader_election.cc:288] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 37e1e5c5ca634982a0a258e89333bf26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7baca47cd2b04cbf9a68e5f0f6305a6d (127.0.98.3:44647), f52a7d693ec54ea19c27916605a799b4 (127.0.98.2:36519)
I20260927 02:41:20.294130 14642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:20.290629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41975 (local address 127.0.98.2:36519)
0927 02:41:20.290762 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:20.290767 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:20.291011 (+ 244us) server_negotiation.cc:407] Connection header received
0927 02:41:20.291159 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:20.291162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:20.291204 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:20.291277 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:41:20.291560 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:20.292054 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:20.293176 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:20.293604 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:20.293683 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:20.293747 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:20.293936 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 02:41:20.294005 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 02:41:20.294130 14639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:20.290625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60523 (local address 127.0.98.3:44647)
0927 02:41:20.290750 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:20.290755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:20.290941 (+ 186us) server_negotiation.cc:407] Connection header received
0927 02:41:20.291060 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:20.291064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:20.291171 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:20.291232 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:41:20.291525 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:20.292033 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:20.293241 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:20.293637 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:20.293735 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:20.293793 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:20.293960 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 02:41:20.294025 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 02:41:20.294142 14652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:20.290512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36519 (local address 127.0.98.1:41975)
0927 02:41:20.290893 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:20.290905 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:41:20.291024 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:41:20.291299 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:41:20.291302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:41:20.291347 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:41:20.291453 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:20.291458 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:20.292158 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:20.292161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:20.293057 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:20.293066 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:20.293723 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:20.293726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:20.293815 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:20.293841 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 02:41:20.293904 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 02:41:20.293974 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":34,"spinlock_wait_cycles":768,"thread_start_us":114,"threads_started":1}
I20260927 02:41:20.294153 14653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:20.290497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44647 (local address 127.0.98.1:60523)
0927 02:41:20.290847 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:20.290858 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:41:20.290959 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:41:20.291264 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:41:20.291267 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:41:20.291282 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:41:20.291419 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:20.291424 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:20.292137 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:20.292140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:20.293111 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:20.293122 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:20.293751 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:20.293754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:20.293850 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:20.293867 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 02:41:20.293936 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 02:41:20.294013 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":69,"spinlock_wait_cycles":23296,"thread_start_us":102,"threads_started":1}
I20260927 02:41:20.294756 14446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99ad8f8adaf24e06bf0ca8ee60abdd01" candidate_uuid: "37e1e5c5ca634982a0a258e89333bf26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f52a7d693ec54ea19c27916605a799b4" is_pre_election: true
I20260927 02:41:20.294925 14580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99ad8f8adaf24e06bf0ca8ee60abdd01" candidate_uuid: "37e1e5c5ca634982a0a258e89333bf26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7baca47cd2b04cbf9a68e5f0f6305a6d" is_pre_election: true
I20260927 02:41:20.295034 14446 raft_consensus.cc:2454] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P f52a7d693ec54ea19c27916605a799b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37e1e5c5ca634982a0a258e89333bf26 in term 0.
I20260927 02:41:20.295207 14580 raft_consensus.cc:2454] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 7baca47cd2b04cbf9a68e5f0f6305a6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37e1e5c5ca634982a0a258e89333bf26 in term 0.
I20260927 02:41:20.295495 14246 leader_election.cc:302] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 37e1e5c5ca634982a0a258e89333bf26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37e1e5c5ca634982a0a258e89333bf26, f52a7d693ec54ea19c27916605a799b4; no voters:
I20260927 02:41:20.295747 14650 raft_consensus.cc:2781] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 37e1e5c5ca634982a0a258e89333bf26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:41:20.295819 14650 raft_consensus.cc:482] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 37e1e5c5ca634982a0a258e89333bf26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:41:20.295849 14650 raft_consensus.cc:3037] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 37e1e5c5ca634982a0a258e89333bf26 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:41:20.296849 14650 raft_consensus.cc:504] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 37e1e5c5ca634982a0a258e89333bf26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7baca47cd2b04cbf9a68e5f0f6305a6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44647 } } peers { permanent_uuid: "37e1e5c5ca634982a0a258e89333bf26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42077 } } peers { permanent_uuid: "f52a7d693ec54ea19c27916605a799b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36519 } }
I20260927 02:41:20.297204 14650 leader_election.cc:288] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 37e1e5c5ca634982a0a258e89333bf26 [CANDIDATE]: Term 1 election: Requested vote from peers 7baca47cd2b04cbf9a68e5f0f6305a6d (127.0.98.3:44647), f52a7d693ec54ea19c27916605a799b4 (127.0.98.2:36519)
I20260927 02:41:20.297577 14580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99ad8f8adaf24e06bf0ca8ee60abdd01" candidate_uuid: "37e1e5c5ca634982a0a258e89333bf26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7baca47cd2b04cbf9a68e5f0f6305a6d"
I20260927 02:41:20.297580 14446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99ad8f8adaf24e06bf0ca8ee60abdd01" candidate_uuid: "37e1e5c5ca634982a0a258e89333bf26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f52a7d693ec54ea19c27916605a799b4"
I20260927 02:41:20.297701 14580 raft_consensus.cc:3037] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 7baca47cd2b04cbf9a68e5f0f6305a6d [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:41:20.297701 14446 raft_consensus.cc:3037] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P f52a7d693ec54ea19c27916605a799b4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:41:20.298997 14580 raft_consensus.cc:2454] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 7baca47cd2b04cbf9a68e5f0f6305a6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37e1e5c5ca634982a0a258e89333bf26 in term 1.
I20260927 02:41:20.299010 14446 raft_consensus.cc:2454] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P f52a7d693ec54ea19c27916605a799b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37e1e5c5ca634982a0a258e89333bf26 in term 1.
I20260927 02:41:20.299299 14245 leader_election.cc:302] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 37e1e5c5ca634982a0a258e89333bf26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37e1e5c5ca634982a0a258e89333bf26, 7baca47cd2b04cbf9a68e5f0f6305a6d; no voters:
I20260927 02:41:20.299485 14650 raft_consensus.cc:2781] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 37e1e5c5ca634982a0a258e89333bf26 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:41:20.299686 14650 raft_consensus.cc:686] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 37e1e5c5ca634982a0a258e89333bf26 [term 1 LEADER]: Becoming Leader. State: Replica: 37e1e5c5ca634982a0a258e89333bf26, State: Running, Role: LEADER
I20260927 02:41:20.300042 14650 consensus_queue.cc:237] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 37e1e5c5ca634982a0a258e89333bf26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7baca47cd2b04cbf9a68e5f0f6305a6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44647 } } peers { permanent_uuid: "37e1e5c5ca634982a0a258e89333bf26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42077 } } peers { permanent_uuid: "f52a7d693ec54ea19c27916605a799b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36519 } }
I20260927 02:41:20.303144 14165 catalog_manager.cc:5495] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 37e1e5c5ca634982a0a258e89333bf26 reported cstate change: term changed from 0 to 1, leader changed from <none> to 37e1e5c5ca634982a0a258e89333bf26 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "37e1e5c5ca634982a0a258e89333bf26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7baca47cd2b04cbf9a68e5f0f6305a6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37e1e5c5ca634982a0a258e89333bf26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f52a7d693ec54ea19c27916605a799b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36519 } health_report { overall_health: UNKNOWN } } }
I20260927 02:41:20.322348 14640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:20.318912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56324 (local address 127.0.98.1:42077)
0927 02:41:20.319032 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:20.319035 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:20.319048 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:41:20.319151 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:20.319153 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:20.319206 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:20.319283 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:41:20.319546 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:20.320059 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:20.320943 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:20.321187 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:20.321267 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:20.321532 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:41:20.321941 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:41:20.322029 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:20.322151 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 02:41:20.322202 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 02:41:20.328342 14446 raft_consensus.cc:1261] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P f52a7d693ec54ea19c27916605a799b4 [term 1 FOLLOWER]: Refusing update from remote peer 37e1e5c5ca634982a0a258e89333bf26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:41:20.328342 14580 raft_consensus.cc:1261] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 7baca47cd2b04cbf9a68e5f0f6305a6d [term 1 FOLLOWER]: Refusing update from remote peer 37e1e5c5ca634982a0a258e89333bf26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:41:20.329021 14650 consensus_queue.cc:1035] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 37e1e5c5ca634982a0a258e89333bf26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f52a7d693ec54ea19c27916605a799b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:41:20.329243 14654 consensus_queue.cc:1035] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 37e1e5c5ca634982a0a258e89333bf26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7baca47cd2b04cbf9a68e5f0f6305a6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 02:41:20.333577 14627 tablet.cc:2369] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 7baca47cd2b04cbf9a68e5f0f6305a6d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:41:20.346643 14166 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:41:20.346820 14166 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:41:20.349419 14166 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997: Sending DeleteTablet for 3 replicas of tablet 99ad8f8adaf24e06bf0ca8ee60abdd01
I20260927 02:41:20.350270 14560 tablet_service.cc:1514] Processing DeleteTablet for tablet 99ad8f8adaf24e06bf0ca8ee60abdd01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:41:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41010
I20260927 02:41:20.350240 14292 tablet_service.cc:1514] Processing DeleteTablet for tablet 99ad8f8adaf24e06bf0ca8ee60abdd01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:41:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56312
I20260927 02:41:20.350435 14426 tablet_service.cc:1514] Processing DeleteTablet for tablet 99ad8f8adaf24e06bf0ca8ee60abdd01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:41:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42650
I20260927 02:41:20.350795 14667 tablet_replica.cc:330] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 37e1e5c5ca634982a0a258e89333bf26: stopping tablet replica
I20260927 02:41:20.351092 14669 tablet_replica.cc:330] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P f52a7d693ec54ea19c27916605a799b4: stopping tablet replica
I20260927 02:41:20.351133 14667 raft_consensus.cc:2229] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 37e1e5c5ca634982a0a258e89333bf26 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:41:20.351161 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 14229
I20260927 02:41:20.352401 14668 tablet_replica.cc:330] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 7baca47cd2b04cbf9a68e5f0f6305a6d: stopping tablet replica
I20260927 02:41:20.352739 14668 raft_consensus.cc:2229] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 7baca47cd2b04cbf9a68e5f0f6305a6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:41:20.352919 14669 raft_consensus.cc:2229] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P f52a7d693ec54ea19c27916605a799b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:41:20.353102 14668 raft_consensus.cc:2258] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 7baca47cd2b04cbf9a68e5f0f6305a6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:41:20.353190 14669 raft_consensus.cc:2258] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P f52a7d693ec54ea19c27916605a799b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:41:20.354514 14669 ts_tablet_manager.cc:1905] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P f52a7d693ec54ea19c27916605a799b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:41:20.355569 14668 ts_tablet_manager.cc:1905] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 7baca47cd2b04cbf9a68e5f0f6305a6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:41:20.356809 14669 ts_tablet_manager.cc:1918] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P f52a7d693ec54ea19c27916605a799b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:41:20.356899 14669 log.cc:1196] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P f52a7d693ec54ea19c27916605a799b4: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-1/wal/wals/99ad8f8adaf24e06bf0ca8ee60abdd01
I20260927 02:41:20.357159 14669 ts_tablet_manager.cc:1939] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P f52a7d693ec54ea19c27916605a799b4: Deleting consensus metadata
W20260927 02:41:20.358040 14152 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42077 (error 108)
I20260927 02:41:20.358106 14668 ts_tablet_manager.cc:1918] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 7baca47cd2b04cbf9a68e5f0f6305a6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:41:20.358187 14668 log.cc:1196] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 7baca47cd2b04cbf9a68e5f0f6305a6d: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-2/wal/wals/99ad8f8adaf24e06bf0ca8ee60abdd01
I20260927 02:41:20.358285 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 14363
I20260927 02:41:20.359211 14637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:20.358961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42077 (local address 127.0.0.1:56334)
0927 02:41:20.359091 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:20.359146 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 02:41:20.359373 14152 catalog_manager.cc:4584] TS 37e1e5c5ca634982a0a258e89333bf26 (127.0.98.1:42077): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 99ad8f8adaf24e06bf0ca8ee60abdd01: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42077: connect: Connection refused (error 111)
I20260927 02:41:20.362428 14668 ts_tablet_manager.cc:1939] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 7baca47cd2b04cbf9a68e5f0f6305a6d: Deleting consensus metadata
I20260927 02:41:20.363550 14151 catalog_manager.cc:4841] TS 7baca47cd2b04cbf9a68e5f0f6305a6d (127.0.98.3:44647): tablet 99ad8f8adaf24e06bf0ca8ee60abdd01 (table test-table [id=2b280bca6f2446aabfcf4053105305d6]) successfully deleted
I20260927 02:41:20.366147 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 14497
I20260927 02:41:20.366508 14637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:20.366224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36519 (local address 127.0.0.1:42662)
0927 02:41:20.366401 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:20.366444 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260927 02:41:20.370913 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 14135
I20260927 02:41:20.375926 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45947
--webserver_interface=127.0.98.62
--webserver_port=32953
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:41:20.481794 14671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:41:20.482064 14671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:41:20.482118 14671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:41:20.486080 14671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:41:20.486143 14671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:41:20.486169 14671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:41:20.486189 14671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:41:20.486209 14671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:41:20.491398 14671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=32953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:41:20.492393 14671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:41:20.493373 14671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:41:20.506060 14671 server_base.cc:1034] running on GCE node
W20260927 02:41:20.505950 14678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:41:20.505941 14677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:41:20.505946 14680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:41:20.506934 14671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:41:20.508056 14671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:41:20.509227 14671 hybrid_clock.cc:648] HybridClock initialized: now 1790476880509204 us; error 34 us; skew 500 ppm
Sep 27 02:41:20 dist-test-slave-zv8z krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476880, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:41:20.511920 14671 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:41:20.512835 14671 webserver.cc:460] Webserver started at http://127.0.98.62:32953/ using document root <none> and password file <none>
I20260927 02:41:20.513370 14671 fs_manager.cc:359] Metadata directory not provided
I20260927 02:41:20.513451 14671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:41:20.517455 14671 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:41:20.519505 14687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:20.520663 14671 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:41:20.520773 14671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "d7409a2403734592bcce742551f20997"
format_stamp: "Formatted at 2026-09-27 02:41:19 on dist-test-slave-zv8z"
server_key: "32528d19bb181df7aa255d8c2a4e30f8"
server_key_iv: "b0beda8656ece766f07094794636c357"
server_key_version: "encryptionkey@0"
I20260927 02:41:20.521355 14671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:41:20.547513 14671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:41:20.550372 14671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:41:20.550556 14671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:41:20.557276 14671 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45947
I20260927 02:41:20.557528 14739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45947 every 8 connection(s)
I20260927 02:41:20.558323 14671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:41:20.561920 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 14671
I20260927 02:41:20.562340 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42077
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45947
--webserver_port=41129
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:41:20.562544 14740 sys_catalog.cc:263] Verifying existing consensus state
I20260927 02:41:20.564119 14740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997: Bootstrap starting.
I20260927 02:41:20.580238 14740 log.cc:824] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997: Log is configured to *not* fsync() on all Append() calls
I20260927 02:41:20.588721 14740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:41:20.589078 14740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997: Bootstrap complete.
I20260927 02:41:20.592939 14740 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7409a2403734592bcce742551f20997" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45947 } }
I20260927 02:41:20.593349 14740 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7409a2403734592bcce742551f20997, State: Initialized, Role: FOLLOWER
I20260927 02:41:20.593811 14740 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7409a2403734592bcce742551f20997" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45947 } }
I20260927 02:41:20.593927 14740 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:41:20.593997 14740 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:41:20.594085 14740 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997 [term 1 FOLLOWER]: Advancing to term 2
I20260927 02:41:20.595971 14740 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7409a2403734592bcce742551f20997" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45947 } }
I20260927 02:41:20.596274 14740 leader_election.cc:302] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7409a2403734592bcce742551f20997; no voters:
I20260927 02:41:20.596560 14740 leader_election.cc:288] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 02:41:20.596657 14744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 02:41:20.596844 14744 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997 [term 2 LEADER]: Becoming Leader. State: Replica: d7409a2403734592bcce742551f20997, State: Running, Role: LEADER
I20260927 02:41:20.597134 14744 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7409a2403734592bcce742551f20997" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45947 } }
I20260927 02:41:20.597769 14740 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:41:20.597998 14744 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7409a2403734592bcce742551f20997. Latest consensus state: current_term: 2 leader_uuid: "d7409a2403734592bcce742551f20997" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7409a2403734592bcce742551f20997" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45947 } } }
I20260927 02:41:20.598112 14744 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997 [sys.catalog]: This master's current role is: LEADER
I20260927 02:41:20.598469 14748 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:41:20.598546 14745 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d7409a2403734592bcce742551f20997" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7409a2403734592bcce742551f20997" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45947 } } }
I20260927 02:41:20.598716 14745 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997 [sys.catalog]: This master's current role is: LEADER
I20260927 02:41:20.602965 14748 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:41:20.603614 14748 catalog_manager.cc:1259] Loaded cluster ID: 113bd70eda5b40158c169ffd208d3f94
I20260927 02:41:20.603667 14748 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:41:20.604817 14748 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 02:41:20.607613 14748 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 02:41:20.612030 14748 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:41:20.612663 14748 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997: Loaded TSK: 0
I20260927 02:41:20.613579 14748 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 02:41:20.683282 14742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:41:20.683545 14742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:41:20.683602 14742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:41:20.687585 14742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:41:20.687641 14742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:41:20.687717 14742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:41:20.692799 14742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42077
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41129
--tserver_master_addrs=127.0.98.62:45947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:41:20.693758 14742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:41:20.694754 14742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:41:20.703998 14770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:41:20.703999 14767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:41:20.704111 14768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:41:20.704303 14742 server_base.cc:1034] running on GCE node
I20260927 02:41:20.704833 14742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:41:20.705472 14742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:41:20.706637 14742 hybrid_clock.cc:648] HybridClock initialized: now 1790476880706624 us; error 29 us; skew 500 ppm
Sep 27 02:41:20 dist-test-slave-zv8z krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476880, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:41:20.709502 14742 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:41:20.710456 14742 webserver.cc:460] Webserver started at http://127.0.98.1:41129/ using document root <none> and password file <none>
I20260927 02:41:20.711006 14742 fs_manager.cc:359] Metadata directory not provided
I20260927 02:41:20.711097 14742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:41:20.714977 14742 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:41:20.717088 14777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:20.718480 14742 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:41:20.718588 14742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-0/wal
uuid: "37e1e5c5ca634982a0a258e89333bf26"
format_stamp: "Formatted at 2026-09-27 02:41:19 on dist-test-slave-zv8z"
server_key: "f6671aefcba165e019246ed6a0575046"
server_key_iv: "80b76de96e47d39b5ce1ae98f3f8539f"
server_key_version: "encryptionkey@0"
I20260927 02:41:20.719137 14742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:41:20.734056 14742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:41:20.737680 14742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:41:20.737879 14742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:41:20.738574 14742 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:41:20.739952 14784 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 02:41:20.744004 14742 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 02:41:20.744078 14742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260927 02:41:20.744139 14742 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 02:41:20.746320 14742 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 02:41:20.746374 14742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260927 02:41:20.746676 14784 tablet_bootstrap.cc:492] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 37e1e5c5ca634982a0a258e89333bf26: Bootstrap starting.
I20260927 02:41:20.756055 14742 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42077
I20260927 02:41:20.756105 14891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42077 every 8 connection(s)
I20260927 02:41:20.756958 14742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:41:20.758458 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 14742
I20260927 02:41:20.758837 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36519
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45947
--webserver_port=44993
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:41:20.766358 14784 log.cc:824] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 37e1e5c5ca634982a0a258e89333bf26: Log is configured to *not* fsync() on all Append() calls
Sep 27 02:41:20 dist-test-slave-zv8z krb5kdc[14122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476880, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:41:20.773519 14896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:20.760315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34067 (local address 127.0.98.62:45947)
0927 02:41:20.760634 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:20.760657 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:20.760700 (+ 43us) server_negotiation.cc:407] Connection header received
0927 02:41:20.761017 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:20.761053 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:20.761476 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:20.761843 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:41:20.762911 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:20.763863 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:20.765020 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:20.765461 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:20.765618 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:20.768151 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:41:20.768158 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:41:20.768172 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:41:20.768205 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:41:20.770216 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:20.770814 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:20.770819 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:20.770824 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:20.770967 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:20.771314 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:20.771316 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:20.771318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:20.771730 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:41:20.771853 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:20.772449 (+ 596us) server_negotiation.cc:299] Negotiation successful
0927 02:41:20.772741 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":89,"threads_started":1}
I20260927 02:41:20.773540 14894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:20.758989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.1:34067)
0927 02:41:20.759590 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:20.759647 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 02:41:20.760553 (+ 906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:41:20.761910 (+ 1357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:41:20.761916 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:41:20.762271 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:41:20.762745 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:20.762759 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:20.764063 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:20.764066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:20.764899 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:20.764908 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:20.765580 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:20.765584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:20.765770 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:20.766330 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:41:20.766346 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:41:20.767955 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:41:20.770345 (+ 2390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:20.770352 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:20.770365 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:20.770696 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:20.771056 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:20.771059 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:20.771061 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:20.771226 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:20.771834 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:41:20.771840 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:41:20.772135 (+ 295us) client_negotiation.cc:769] Sending connection context
0927 02:41:20.772320 (+ 185us) client_negotiation.cc:240] Negotiation successful
0927 02:41:20.772631 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":174,"threads_started":1}
I20260927 02:41:20.774734 14784 tablet_bootstrap.cc:492] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 37e1e5c5ca634982a0a258e89333bf26: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:41:20.775063 14784 tablet_bootstrap.cc:492] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 37e1e5c5ca634982a0a258e89333bf26: Bootstrap complete.
I20260927 02:41:20.776233 14784 ts_tablet_manager.cc:1397] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 37e1e5c5ca634982a0a258e89333bf26: Time spent bootstrapping tablet: real 0.030s user 0.022s sys 0.004s
I20260927 02:41:20.776291 14892 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45947
I20260927 02:41:20.776530 14892 heartbeater.cc:464] Registering TS with master...
I20260927 02:41:20.777051 14892 heartbeater.cc:511] Master 127.0.98.62:45947 requested a full tablet report, sending...
I20260927 02:41:20.778357 14784 raft_consensus.cc:348] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 37e1e5c5ca634982a0a258e89333bf26 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7baca47cd2b04cbf9a68e5f0f6305a6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44647 } } peers { permanent_uuid: "37e1e5c5ca634982a0a258e89333bf26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42077 } } peers { permanent_uuid: "f52a7d693ec54ea19c27916605a799b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36519 } }
I20260927 02:41:20.778820 14784 raft_consensus.cc:729] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 37e1e5c5ca634982a0a258e89333bf26 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37e1e5c5ca634982a0a258e89333bf26, State: Initialized, Role: FOLLOWER
I20260927 02:41:20.779285 14784 consensus_queue.cc:260] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 37e1e5c5ca634982a0a258e89333bf26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7baca47cd2b04cbf9a68e5f0f6305a6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44647 } } peers { permanent_uuid: "37e1e5c5ca634982a0a258e89333bf26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42077 } } peers { permanent_uuid: "f52a7d693ec54ea19c27916605a799b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36519 } }
I20260927 02:41:20.779627 14704 ts_manager.cc:194] Registered new tserver with Master: 37e1e5c5ca634982a0a258e89333bf26 (127.0.98.1:42077)
I20260927 02:41:20.780275 14784 ts_tablet_manager.cc:1428] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 37e1e5c5ca634982a0a258e89333bf26: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260927 02:41:20.782977 14704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34067
I20260927 02:41:20.785590 14892 heartbeater.cc:503] Master 127.0.98.62:45947 was elected leader, sending a full tablet report...
Sep 27 02:41:20 dist-test-slave-zv8z krb5kdc[14122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476880, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:41:20.792423 14899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:20.782401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42077 (local address 127.0.0.1:48260)
0927 02:41:20.783093 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:20.783135 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 02:41:20.783266 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:41:20.783757 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:41:20.783763 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:41:20.784151 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:41:20.784279 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:20.784286 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:20.785281 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:20.785285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:20.785961 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:20.785973 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:20.786724 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:20.786742 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:20.786855 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:20.787468 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:41:20.787481 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:41:20.788922 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:41:20.790893 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:20.790900 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:20.790905 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:20.791125 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:20.791398 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:20.791400 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:20.791401 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:20.791469 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:20.791904 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:41:20.791908 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:41:20.792038 (+ 130us) client_negotiation.cc:769] Sending connection context
0927 02:41:20.792147 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 02:41:20.792293 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":121,"threads_started":1}
I20260927 02:41:20.792776 14900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:20.782489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48260 (local address 127.0.98.1:42077)
0927 02:41:20.782841 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:20.782853 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:20.783218 (+ 365us) server_negotiation.cc:407] Connection header received
0927 02:41:20.783404 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:20.783409 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:20.783609 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:20.783740 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:41:20.784390 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:20.785171 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:20.786094 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:20.786618 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:20.786722 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:20.789058 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:41:20.789067 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:41:20.789080 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:41:20.789106 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:41:20.790796 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:20.791223 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:20.791229 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:20.791231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:20.791320 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:20.791542 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:20.791545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:20.791546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:20.791815 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:41:20.791933 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:20.792237 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 02:41:20.792399 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":108,"threads_started":1}
I20260927 02:41:20.793691 14824 tablet_service.cc:1514] Processing DeleteTablet for tablet 99ad8f8adaf24e06bf0ca8ee60abdd01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:41:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48260
I20260927 02:41:20.794539 14901 tablet_replica.cc:330] stopping tablet replica
I20260927 02:41:20.794813 14901 raft_consensus.cc:2229] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 37e1e5c5ca634982a0a258e89333bf26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:41:20.795046 14901 raft_consensus.cc:2258] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 37e1e5c5ca634982a0a258e89333bf26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:41:20.795953 14901 ts_tablet_manager.cc:1905] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 37e1e5c5ca634982a0a258e89333bf26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:41:20.799387 14901 ts_tablet_manager.cc:1918] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 37e1e5c5ca634982a0a258e89333bf26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:41:20.799465 14901 log.cc:1196] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 37e1e5c5ca634982a0a258e89333bf26: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-0/wal/wals/99ad8f8adaf24e06bf0ca8ee60abdd01
I20260927 02:41:20.799698 14901 ts_tablet_manager.cc:1939] T 99ad8f8adaf24e06bf0ca8ee60abdd01 P 37e1e5c5ca634982a0a258e89333bf26: Deleting consensus metadata
I20260927 02:41:20.801278 14689 catalog_manager.cc:4841] TS 37e1e5c5ca634982a0a258e89333bf26 (127.0.98.1:42077): tablet 99ad8f8adaf24e06bf0ca8ee60abdd01 (table test-table [id=2b280bca6f2446aabfcf4053105305d6]) successfully deleted
W20260927 02:41:20.875034 14895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:41:20.875294 14895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:41:20.875355 14895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:41:20.879340 14895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:41:20.879395 14895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:41:20.879472 14895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:41:20.884469 14895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36519
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44993
--tserver_master_addrs=127.0.98.62:45947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:41:20.885421 14895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:41:20.886435 14895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:41:20.895262 14906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:41:20.895407 14895 server_base.cc:1034] running on GCE node
W20260927 02:41:20.895280 14909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:41:20.895298 14907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:41:20.895941 14895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:41:20.896477 14895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:41:20.897626 14895 hybrid_clock.cc:648] HybridClock initialized: now 1790476880897601 us; error 41 us; skew 500 ppm
Sep 27 02:41:20 dist-test-slave-zv8z krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476880, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:41:20.900239 14895 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:41:20.901034 14895 webserver.cc:460] Webserver started at http://127.0.98.2:44993/ using document root <none> and password file <none>
I20260927 02:41:20.901520 14895 fs_manager.cc:359] Metadata directory not provided
I20260927 02:41:20.901590 14895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:41:20.905499 14895 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 02:41:20.907778 14916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:20.909011 14895 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:41:20.909142 14895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-1/wal
uuid: "f52a7d693ec54ea19c27916605a799b4"
format_stamp: "Formatted at 2026-09-27 02:41:19 on dist-test-slave-zv8z"
server_key: "a0c4d48df42b09f807c9e078eb24d97a"
server_key_iv: "2e69127122017f700a318d37feb1df45"
server_key_version: "encryptionkey@0"
I20260927 02:41:20.909677 14895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:41:20.922960 14895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:41:20.925737 14895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:41:20.925894 14895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:41:20.926486 14895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:41:20.927263 14895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:41:20.927309 14895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:20.927357 14895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:41:20.927385 14895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:20.935441 14895 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36519
I20260927 02:41:20.935475 15029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36519 every 8 connection(s)
I20260927 02:41:20.936370 14895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 02:41:20 dist-test-slave-zv8z krb5kdc[14122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476880, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:41:20.946517 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 14895
I20260927 02:41:20.946864 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44647
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45947
--webserver_port=40909
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:41:20.948876 14896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:20.937944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52453 (local address 127.0.98.62:45947)
0927 02:41:20.938063 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:20.938067 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:20.938892 (+ 825us) server_negotiation.cc:407] Connection header received
0927 02:41:20.940120 (+ 1228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:20.940123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:20.940186 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:20.940254 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:41:20.941371 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:20.941893 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:20.943011 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:20.943242 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:20.943330 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:20.945639 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:41:20.945642 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:41:20.945644 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:41:20.945674 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:41:20.946946 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:20.947447 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:20.947450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:20.947452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:20.947530 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:20.947829 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:20.947833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:20.947834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:20.948021 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:41:20.948104 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:20.948585 (+ 481us) server_negotiation.cc:299] Negotiation successful
0927 02:41:20.948714 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 02:41:20.949689 15032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:20.938340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.2:52453)
0927 02:41:20.938754 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:20.938793 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:41:20.939911 (+ 1118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:41:20.940435 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:41:20.940441 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:41:20.940771 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:41:20.941224 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:20.941238 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:20.942014 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:20.942017 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:20.942887 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:20.942896 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:20.943377 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:20.943381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:20.943541 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:20.944063 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:41:20.944079 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:41:20.945453 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:41:20.947099 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:20.947105 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:20.947120 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:20.947353 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:20.947610 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:20.947613 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:20.947615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:20.947742 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:20.948118 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:41:20.948125 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:41:20.948361 (+ 236us) client_negotiation.cc:769] Sending connection context
0927 02:41:20.948534 (+ 173us) client_negotiation.cc:240] Negotiation successful
0927 02:41:20.948857 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":107,"threads_started":1}
I20260927 02:41:20.950865 15030 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45947
I20260927 02:41:20.951121 15030 heartbeater.cc:464] Registering TS with master...
I20260927 02:41:20.951602 15030 heartbeater.cc:511] Master 127.0.98.62:45947 requested a full tablet report, sending...
I20260927 02:41:20.952606 14704 ts_manager.cc:194] Registered new tserver with Master: f52a7d693ec54ea19c27916605a799b4 (127.0.98.2:36519)
I20260927 02:41:20.953154 14704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52453
W20260927 02:41:21.055060 15033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:41:21.055333 15033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:41:21.055394 15033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:41:21.059427 15033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:41:21.059495 15033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:41:21.059577 15033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:41:21.064904 15033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44647
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40909
--tserver_master_addrs=127.0.98.62:45947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:41:21.065989 15033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:41:21.067040 15033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:41:21.076469 15038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:41:21.076472 15039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:41:21.076472 15041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:41:21.076679 15033 server_base.cc:1034] running on GCE node
I20260927 02:41:21.077302 15033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:41:21.077886 15033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:41:21.079043 15033 hybrid_clock.cc:648] HybridClock initialized: now 1790476881079024 us; error 27 us; skew 500 ppm
Sep 27 02:41:21 dist-test-slave-zv8z krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:41:21.081609 15033 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:41:21.082530 15033 webserver.cc:460] Webserver started at http://127.0.98.3:40909/ using document root <none> and password file <none>
I20260927 02:41:21.083050 15033 fs_manager.cc:359] Metadata directory not provided
I20260927 02:41:21.083119 15033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:41:21.086802 15033 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:41:21.088792 15048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:21.089982 15033 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:41:21.090093 15033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-2/wal
uuid: "7baca47cd2b04cbf9a68e5f0f6305a6d"
format_stamp: "Formatted at 2026-09-27 02:41:20 on dist-test-slave-zv8z"
server_key: "bfdc1c2c2d5f6176ee1e7fc64974b164"
server_key_iv: "9f41d3d33bab1866beaa4909769508f3"
server_key_version: "encryptionkey@0"
I20260927 02:41:21.090658 15033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:41:21.108634 15033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:41:21.111986 15033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:41:21.112171 15033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:41:21.112820 15033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:41:21.113665 15033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:41:21.113715 15033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:21.113767 15033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:41:21.113790 15033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:21.121979 15033 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44647
I20260927 02:41:21.122021 15161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44647 every 8 connection(s)
I20260927 02:41:21.123049 15033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 02:41:21 dist-test-slave-zv8z krb5kdc[14122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:41:21.133155 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 15033
I20260927 02:41:21.136260 14896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:21.124865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55255 (local address 127.0.98.62:45947)
0927 02:41:21.125002 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:21.125007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:21.125873 (+ 866us) server_negotiation.cc:407] Connection header received
0927 02:41:21.127165 (+ 1292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:21.127168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:21.127214 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:21.127289 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:41:21.128380 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:21.128884 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:21.130033 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:21.130300 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:21.130396 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:21.132794 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:41:21.132798 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:41:21.132800 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:41:21.132825 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:41:21.134394 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:21.134842 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:21.134845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:21.134847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:21.134923 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:21.135228 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:21.135231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:21.135232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:21.135427 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:41:21.135509 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:21.135982 (+ 473us) server_negotiation.cc:299] Negotiation successful
0927 02:41:21.136099 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":43}
I20260927 02:41:21.137095 15164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:21.125304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.3:55255)
0927 02:41:21.125731 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:21.125773 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 02:41:21.126956 (+ 1183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:41:21.127445 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:41:21.127453 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:41:21.127790 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:41:21.128255 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:21.128268 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:21.128971 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:21.128973 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:21.129865 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:21.129873 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:21.130438 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:21.130442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:21.130584 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:21.131124 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:41:21.131140 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:41:21.132638 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:41:21.134494 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:21.134500 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:21.134515 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:21.134754 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:21.135008 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:21.135011 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:21.135013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:21.135145 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:21.135522 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:41:21.135528 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:41:21.135759 (+ 231us) client_negotiation.cc:769] Sending connection context
0927 02:41:21.135920 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 02:41:21.136232 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":102,"threads_started":1}
I20260927 02:41:21.138167 15162 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45947
I20260927 02:41:21.138411 15162 heartbeater.cc:464] Registering TS with master...
I20260927 02:41:21.138937 15162 heartbeater.cc:511] Master 127.0.98.62:45947 requested a full tablet report, sending...
I20260927 02:41:21.139916 14704 ts_manager.cc:194] Registered new tserver with Master: 7baca47cd2b04cbf9a68e5f0f6305a6d (127.0.98.3:44647)
I20260927 02:41:21.140439 14704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55255
I20260927 02:41:21.141249 15165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:21.133745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48670 (local address 127.0.98.62:45947)
0927 02:41:21.133962 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:21.133965 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:21.134357 (+ 392us) server_negotiation.cc:407] Connection header received
0927 02:41:21.134403 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:21.134405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:21.134444 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:21.134515 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:41:21.135188 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:21.135758 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:21.136674 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:21.136902 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:21.136982 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:21.138303 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:41:21.138306 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:41:21.138309 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:41:21.138333 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:41:21.139715 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:21.140096 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:21.140100 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:21.140102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:21.140180 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:21.140414 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:21.140417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:21.140419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:21.140599 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:41:21.140772 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:21.140940 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 02:41:21.141098 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":151,"thread_start_us":63,"threads_started":1}
I20260927 02:41:21.143023 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:41:21.147500 15165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:21.143660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48682 (local address 127.0.98.62:45947)
0927 02:41:21.143842 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:21.143845 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:21.144003 (+ 158us) server_negotiation.cc:407] Connection header received
0927 02:41:21.144078 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:21.144080 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:21.144170 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:21.144255 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:41:21.144540 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:21.145077 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:21.146016 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:21.146268 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:21.146360 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:21.146687 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:41:21.147117 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:41:21.147213 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:21.147322 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 02:41:21.147380 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 02:41:21.148231 14704 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:48682:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:41:21.151809 14704 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:41:21.159400 14824 tablet_service.cc:1467] Processing CreateTablet for tablet 2c3aae985f0a4178a1f84f9f66898697 (DEFAULT_TABLE table=test-table [id=c36cc53479f34fb380f01a7a69f53a00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:41:21.160079 14824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c3aae985f0a4178a1f84f9f66898697. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:41:21.162858 14784 tablet_bootstrap.cc:492] T 2c3aae985f0a4178a1f84f9f66898697 P 37e1e5c5ca634982a0a258e89333bf26: Bootstrap starting.
I20260927 02:41:21.163293 14899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:21.158154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44647 (local address 127.0.0.1:59312)
0927 02:41:21.158402 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:21.158423 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 02:41:21.158507 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:41:21.159403 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:41:21.159406 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:41:21.159424 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:41:21.159535 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:21.159541 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:21.161088 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:21.161092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:21.161989 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:21.161998 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:21.162887 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:21.162891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:21.162990 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:21.163040 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 02:41:21.163089 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 02:41:21.163167 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260927 02:41:21.163729 14784 tablet_bootstrap.cc:654] T 2c3aae985f0a4178a1f84f9f66898697 P 37e1e5c5ca634982a0a258e89333bf26: Neither blocks nor log segments found. Creating new log.
I20260927 02:41:21.163899 15172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:21.158519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36519 (local address 127.0.0.1:60674)
0927 02:41:21.159029 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:21.159043 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:41:21.159126 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:41:21.159787 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:41:21.159790 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:41:21.159808 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:41:21.159894 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:21.159900 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:21.161384 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:21.161387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:21.162543 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:21.162554 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:21.163571 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:21.163574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:21.163657 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:21.163678 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 02:41:21.163723 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 02:41:21.163784 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":76,"threads_started":1}
I20260927 02:41:21.163913 15171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:21.158403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59312 (local address 127.0.98.3:44647)
0927 02:41:21.158770 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:21.158902 (+ 132us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:21.158940 (+ 38us) server_negotiation.cc:407] Connection header received
0927 02:41:21.159020 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:21.159025 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:21.159249 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:21.159418 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:41:21.159666 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:21.160950 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:21.162113 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:21.162759 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:21.163475 (+ 716us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:21.163629 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:21.163680 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 02:41:21.163773 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":182,"threads_started":1}
I20260927 02:41:21.164561 15173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:21.159030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60674 (local address 127.0.98.2:36519)
0927 02:41:21.159341 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:21.159353 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:21.159375 (+ 22us) server_negotiation.cc:407] Connection header received
0927 02:41:21.159442 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:21.159448 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:21.159650 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:21.159815 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:41:21.160058 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:21.161272 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:21.162669 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:21.163448 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:21.164156 (+ 708us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:21.164289 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:21.164332 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 02:41:21.164411 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":108,"threads_started":1}
I20260927 02:41:21.165132 14784 tablet_bootstrap.cc:492] T 2c3aae985f0a4178a1f84f9f66898697 P 37e1e5c5ca634982a0a258e89333bf26: No bootstrap required, opened a new log
I20260927 02:41:21.165210 14784 ts_tablet_manager.cc:1397] T 2c3aae985f0a4178a1f84f9f66898697 P 37e1e5c5ca634982a0a258e89333bf26: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:41:21.165639 14784 raft_consensus.cc:348] T 2c3aae985f0a4178a1f84f9f66898697 P 37e1e5c5ca634982a0a258e89333bf26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7baca47cd2b04cbf9a68e5f0f6305a6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44647 } } peers { permanent_uuid: "f52a7d693ec54ea19c27916605a799b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36519 } } peers { permanent_uuid: "37e1e5c5ca634982a0a258e89333bf26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42077 } }
I20260927 02:41:21.165753 14784 raft_consensus.cc:374] T 2c3aae985f0a4178a1f84f9f66898697 P 37e1e5c5ca634982a0a258e89333bf26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:41:21.165788 14784 raft_consensus.cc:729] T 2c3aae985f0a4178a1f84f9f66898697 P 37e1e5c5ca634982a0a258e89333bf26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37e1e5c5ca634982a0a258e89333bf26, State: Initialized, Role: FOLLOWER
I20260927 02:41:21.165907 14784 consensus_queue.cc:260] T 2c3aae985f0a4178a1f84f9f66898697 P 37e1e5c5ca634982a0a258e89333bf26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7baca47cd2b04cbf9a68e5f0f6305a6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44647 } } peers { permanent_uuid: "f52a7d693ec54ea19c27916605a799b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36519 } } peers { permanent_uuid: "37e1e5c5ca634982a0a258e89333bf26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42077 } }
I20260927 02:41:21.166159 14784 ts_tablet_manager.cc:1428] T 2c3aae985f0a4178a1f84f9f66898697 P 37e1e5c5ca634982a0a258e89333bf26: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:41:21.166874 15096 tablet_service.cc:1467] Processing CreateTablet for tablet 2c3aae985f0a4178a1f84f9f66898697 (DEFAULT_TABLE table=test-table [id=c36cc53479f34fb380f01a7a69f53a00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:41:21.167316 14964 tablet_service.cc:1467] Processing CreateTablet for tablet 2c3aae985f0a4178a1f84f9f66898697 (DEFAULT_TABLE table=test-table [id=c36cc53479f34fb380f01a7a69f53a00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:41:21.167860 15096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c3aae985f0a4178a1f84f9f66898697. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:41:21.168286 14964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c3aae985f0a4178a1f84f9f66898697. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:41:21.174319 15175 tablet_bootstrap.cc:492] T 2c3aae985f0a4178a1f84f9f66898697 P 7baca47cd2b04cbf9a68e5f0f6305a6d: Bootstrap starting.
I20260927 02:41:21.174222 15176 tablet_bootstrap.cc:492] T 2c3aae985f0a4178a1f84f9f66898697 P f52a7d693ec54ea19c27916605a799b4: Bootstrap starting.
I20260927 02:41:21.176865 15176 tablet_bootstrap.cc:654] T 2c3aae985f0a4178a1f84f9f66898697 P f52a7d693ec54ea19c27916605a799b4: Neither blocks nor log segments found. Creating new log.
I20260927 02:41:21.177559 15176 log.cc:824] T 2c3aae985f0a4178a1f84f9f66898697 P f52a7d693ec54ea19c27916605a799b4: Log is configured to *not* fsync() on all Append() calls
I20260927 02:41:21.178663 15175 tablet_bootstrap.cc:654] T 2c3aae985f0a4178a1f84f9f66898697 P 7baca47cd2b04cbf9a68e5f0f6305a6d: Neither blocks nor log segments found. Creating new log.
I20260927 02:41:21.179387 15176 tablet_bootstrap.cc:492] T 2c3aae985f0a4178a1f84f9f66898697 P f52a7d693ec54ea19c27916605a799b4: No bootstrap required, opened a new log
I20260927 02:41:21.179591 15176 ts_tablet_manager.cc:1397] T 2c3aae985f0a4178a1f84f9f66898697 P f52a7d693ec54ea19c27916605a799b4: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260927 02:41:21.179808 15175 log.cc:824] T 2c3aae985f0a4178a1f84f9f66898697 P 7baca47cd2b04cbf9a68e5f0f6305a6d: Log is configured to *not* fsync() on all Append() calls
I20260927 02:41:21.181511 15175 tablet_bootstrap.cc:492] T 2c3aae985f0a4178a1f84f9f66898697 P 7baca47cd2b04cbf9a68e5f0f6305a6d: No bootstrap required, opened a new log
I20260927 02:41:21.181686 15175 ts_tablet_manager.cc:1397] T 2c3aae985f0a4178a1f84f9f66898697 P 7baca47cd2b04cbf9a68e5f0f6305a6d: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260927 02:41:21.182668 15176 raft_consensus.cc:348] T 2c3aae985f0a4178a1f84f9f66898697 P f52a7d693ec54ea19c27916605a799b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7baca47cd2b04cbf9a68e5f0f6305a6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44647 } } peers { permanent_uuid: "f52a7d693ec54ea19c27916605a799b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36519 } } peers { permanent_uuid: "37e1e5c5ca634982a0a258e89333bf26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42077 } }
I20260927 02:41:21.182894 15176 raft_consensus.cc:374] T 2c3aae985f0a4178a1f84f9f66898697 P f52a7d693ec54ea19c27916605a799b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:41:21.182927 15176 raft_consensus.cc:729] T 2c3aae985f0a4178a1f84f9f66898697 P f52a7d693ec54ea19c27916605a799b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f52a7d693ec54ea19c27916605a799b4, State: Initialized, Role: FOLLOWER
I20260927 02:41:21.183362 15176 consensus_queue.cc:260] T 2c3aae985f0a4178a1f84f9f66898697 P f52a7d693ec54ea19c27916605a799b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7baca47cd2b04cbf9a68e5f0f6305a6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44647 } } peers { permanent_uuid: "f52a7d693ec54ea19c27916605a799b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36519 } } peers { permanent_uuid: "37e1e5c5ca634982a0a258e89333bf26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42077 } }
I20260927 02:41:21.184013 15030 heartbeater.cc:503] Master 127.0.98.62:45947 was elected leader, sending a full tablet report...
I20260927 02:41:21.184360 15176 ts_tablet_manager.cc:1428] T 2c3aae985f0a4178a1f84f9f66898697 P f52a7d693ec54ea19c27916605a799b4: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 02:41:21.185043 15175 raft_consensus.cc:348] T 2c3aae985f0a4178a1f84f9f66898697 P 7baca47cd2b04cbf9a68e5f0f6305a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7baca47cd2b04cbf9a68e5f0f6305a6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44647 } } peers { permanent_uuid: "f52a7d693ec54ea19c27916605a799b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36519 } } peers { permanent_uuid: "37e1e5c5ca634982a0a258e89333bf26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42077 } }
I20260927 02:41:21.185328 15175 raft_consensus.cc:374] T 2c3aae985f0a4178a1f84f9f66898697 P 7baca47cd2b04cbf9a68e5f0f6305a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:41:21.185395 15175 raft_consensus.cc:729] T 2c3aae985f0a4178a1f84f9f66898697 P 7baca47cd2b04cbf9a68e5f0f6305a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7baca47cd2b04cbf9a68e5f0f6305a6d, State: Initialized, Role: FOLLOWER
I20260927 02:41:21.185817 15175 consensus_queue.cc:260] T 2c3aae985f0a4178a1f84f9f66898697 P 7baca47cd2b04cbf9a68e5f0f6305a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7baca47cd2b04cbf9a68e5f0f6305a6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44647 } } peers { permanent_uuid: "f52a7d693ec54ea19c27916605a799b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36519 } } peers { permanent_uuid: "37e1e5c5ca634982a0a258e89333bf26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42077 } }
I20260927 02:41:21.186873 15175 ts_tablet_manager.cc:1428] T 2c3aae985f0a4178a1f84f9f66898697 P 7baca47cd2b04cbf9a68e5f0f6305a6d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 02:41:21.187111 15162 heartbeater.cc:503] Master 127.0.98.62:45947 was elected leader, sending a full tablet report...
W20260927 02:41:21.187188 15031 tablet.cc:2369] T 2c3aae985f0a4178a1f84f9f66898697 P f52a7d693ec54ea19c27916605a799b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:41:21.258186 14893 tablet.cc:2369] T 2c3aae985f0a4178a1f84f9f66898697 P 37e1e5c5ca634982a0a258e89333bf26: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:41:21.374100 15163 tablet.cc:2369] T 2c3aae985f0a4178a1f84f9f66898697 P 7baca47cd2b04cbf9a68e5f0f6305a6d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:41:21.379714 15180 raft_consensus.cc:482] T 2c3aae985f0a4178a1f84f9f66898697 P 7baca47cd2b04cbf9a68e5f0f6305a6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:41:21.379979 15180 raft_consensus.cc:504] T 2c3aae985f0a4178a1f84f9f66898697 P 7baca47cd2b04cbf9a68e5f0f6305a6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7baca47cd2b04cbf9a68e5f0f6305a6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44647 } } peers { permanent_uuid: "f52a7d693ec54ea19c27916605a799b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36519 } } peers { permanent_uuid: "37e1e5c5ca634982a0a258e89333bf26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42077 } }
I20260927 02:41:21.381104 15180 leader_election.cc:288] T 2c3aae985f0a4178a1f84f9f66898697 P 7baca47cd2b04cbf9a68e5f0f6305a6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f52a7d693ec54ea19c27916605a799b4 (127.0.98.2:36519), 37e1e5c5ca634982a0a258e89333bf26 (127.0.98.1:42077)
I20260927 02:41:21.384768 15173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:21.381538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53743 (local address 127.0.98.2:36519)
0927 02:41:21.381698 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:21.381703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:21.381719 (+ 16us) server_negotiation.cc:407] Connection header received
0927 02:41:21.381780 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:21.381783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:21.381838 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:21.381925 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:41:21.382227 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:21.382755 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:21.383840 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:21.384246 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:21.384350 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:21.384417 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:21.384581 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 02:41:21.384653 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 02:41:21.384768 15164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:21.381405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36519 (local address 127.0.98.3:53743)
0927 02:41:21.381528 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:21.381544 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 02:41:21.381656 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:41:21.381999 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:41:21.382002 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:41:21.382022 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:41:21.382116 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:21.382121 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:21.382864 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:21.382867 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:21.383706 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:21.383716 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:21.384359 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:21.384363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:21.384469 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:21.384487 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 02:41:21.384553 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 02:41:21.384632 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 02:41:21.385226 15181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:21.381652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42077 (local address 127.0.98.3:55397)
0927 02:41:21.382042 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:21.382055 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:41:21.382152 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:41:21.382461 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:41:21.382463 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:41:21.382479 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:41:21.382594 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:21.382599 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:21.383326 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:21.383329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:21.384105 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:21.384113 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:21.384823 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:21.384826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:21.384937 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:21.384961 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 02:41:21.385019 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 02:41:21.385084 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":111,"threads_started":1}
I20260927 02:41:21.385332 15182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:21.381785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55397 (local address 127.0.98.1:42077)
0927 02:41:21.382044 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:21.382048 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:21.382136 (+ 88us) server_negotiation.cc:407] Connection header received
0927 02:41:21.382273 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:21.382276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:21.382340 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:21.382439 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:41:21.382703 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:21.383216 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:21.384232 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:21.384692 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:21.384789 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:21.385099 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:21.385142 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 02:41:21.385215 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":84,"threads_started":1}
I20260927 02:41:21.385474 14984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3aae985f0a4178a1f84f9f66898697" candidate_uuid: "7baca47cd2b04cbf9a68e5f0f6305a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f52a7d693ec54ea19c27916605a799b4" is_pre_election: true
I20260927 02:41:21.385833 14984 raft_consensus.cc:2454] T 2c3aae985f0a4178a1f84f9f66898697 P f52a7d693ec54ea19c27916605a799b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7baca47cd2b04cbf9a68e5f0f6305a6d in term 0.
I20260927 02:41:21.385797 14846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3aae985f0a4178a1f84f9f66898697" candidate_uuid: "7baca47cd2b04cbf9a68e5f0f6305a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37e1e5c5ca634982a0a258e89333bf26" is_pre_election: true
I20260927 02:41:21.386063 14846 raft_consensus.cc:2454] T 2c3aae985f0a4178a1f84f9f66898697 P 37e1e5c5ca634982a0a258e89333bf26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7baca47cd2b04cbf9a68e5f0f6305a6d in term 0.
I20260927 02:41:21.386343 15050 leader_election.cc:302] T 2c3aae985f0a4178a1f84f9f66898697 P 7baca47cd2b04cbf9a68e5f0f6305a6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7baca47cd2b04cbf9a68e5f0f6305a6d, f52a7d693ec54ea19c27916605a799b4; no voters:
I20260927 02:41:21.386616 15180 raft_consensus.cc:2781] T 2c3aae985f0a4178a1f84f9f66898697 P 7baca47cd2b04cbf9a68e5f0f6305a6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:41:21.386693 15180 raft_consensus.cc:482] T 2c3aae985f0a4178a1f84f9f66898697 P 7baca47cd2b04cbf9a68e5f0f6305a6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:41:21.386739 15180 raft_consensus.cc:3037] T 2c3aae985f0a4178a1f84f9f66898697 P 7baca47cd2b04cbf9a68e5f0f6305a6d [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:41:21.387741 15180 raft_consensus.cc:504] T 2c3aae985f0a4178a1f84f9f66898697 P 7baca47cd2b04cbf9a68e5f0f6305a6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7baca47cd2b04cbf9a68e5f0f6305a6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44647 } } peers { permanent_uuid: "f52a7d693ec54ea19c27916605a799b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36519 } } peers { permanent_uuid: "37e1e5c5ca634982a0a258e89333bf26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42077 } }
I20260927 02:41:21.388069 15180 leader_election.cc:288] T 2c3aae985f0a4178a1f84f9f66898697 P 7baca47cd2b04cbf9a68e5f0f6305a6d [CANDIDATE]: Term 1 election: Requested vote from peers f52a7d693ec54ea19c27916605a799b4 (127.0.98.2:36519), 37e1e5c5ca634982a0a258e89333bf26 (127.0.98.1:42077)
I20260927 02:41:21.388464 14984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3aae985f0a4178a1f84f9f66898697" candidate_uuid: "7baca47cd2b04cbf9a68e5f0f6305a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f52a7d693ec54ea19c27916605a799b4"
I20260927 02:41:21.388545 14846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3aae985f0a4178a1f84f9f66898697" candidate_uuid: "7baca47cd2b04cbf9a68e5f0f6305a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37e1e5c5ca634982a0a258e89333bf26"
I20260927 02:41:21.388592 14984 raft_consensus.cc:3037] T 2c3aae985f0a4178a1f84f9f66898697 P f52a7d693ec54ea19c27916605a799b4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:41:21.388687 14846 raft_consensus.cc:3037] T 2c3aae985f0a4178a1f84f9f66898697 P 37e1e5c5ca634982a0a258e89333bf26 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:41:21.389660 14846 raft_consensus.cc:2454] T 2c3aae985f0a4178a1f84f9f66898697 P 37e1e5c5ca634982a0a258e89333bf26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7baca47cd2b04cbf9a68e5f0f6305a6d in term 1.
I20260927 02:41:21.389665 14984 raft_consensus.cc:2454] T 2c3aae985f0a4178a1f84f9f66898697 P f52a7d693ec54ea19c27916605a799b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7baca47cd2b04cbf9a68e5f0f6305a6d in term 1.
I20260927 02:41:21.390003 15050 leader_election.cc:302] T 2c3aae985f0a4178a1f84f9f66898697 P 7baca47cd2b04cbf9a68e5f0f6305a6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7baca47cd2b04cbf9a68e5f0f6305a6d, f52a7d693ec54ea19c27916605a799b4; no voters:
I20260927 02:41:21.390194 15180 raft_consensus.cc:2781] T 2c3aae985f0a4178a1f84f9f66898697 P 7baca47cd2b04cbf9a68e5f0f6305a6d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:41:21.390441 15180 raft_consensus.cc:686] T 2c3aae985f0a4178a1f84f9f66898697 P 7baca47cd2b04cbf9a68e5f0f6305a6d [term 1 LEADER]: Becoming Leader. State: Replica: 7baca47cd2b04cbf9a68e5f0f6305a6d, State: Running, Role: LEADER
I20260927 02:41:21.390774 15180 consensus_queue.cc:237] T 2c3aae985f0a4178a1f84f9f66898697 P 7baca47cd2b04cbf9a68e5f0f6305a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7baca47cd2b04cbf9a68e5f0f6305a6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44647 } } peers { permanent_uuid: "f52a7d693ec54ea19c27916605a799b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36519 } } peers { permanent_uuid: "37e1e5c5ca634982a0a258e89333bf26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42077 } }
I20260927 02:41:21.393721 14703 catalog_manager.cc:5495] T 2c3aae985f0a4178a1f84f9f66898697 P 7baca47cd2b04cbf9a68e5f0f6305a6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 7baca47cd2b04cbf9a68e5f0f6305a6d (127.0.98.3). New cstate: current_term: 1 leader_uuid: "7baca47cd2b04cbf9a68e5f0f6305a6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7baca47cd2b04cbf9a68e5f0f6305a6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f52a7d693ec54ea19c27916605a799b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37e1e5c5ca634982a0a258e89333bf26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42077 } health_report { overall_health: UNKNOWN } } }
I20260927 02:41:21.404356 15171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:21.400982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59314 (local address 127.0.98.3:44647)
0927 02:41:21.401097 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:21.401102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:21.401115 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:41:21.401264 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:21.401267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:21.401316 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:21.401400 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:41:21.401638 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:21.402128 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:21.402993 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:21.403198 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:21.403279 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:21.403571 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:41:21.403975 (+ 404us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:41:21.404076 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:21.404181 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 02:41:21.404232 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 02:41:21.409945 14984 raft_consensus.cc:1261] T 2c3aae985f0a4178a1f84f9f66898697 P f52a7d693ec54ea19c27916605a799b4 [term 1 FOLLOWER]: Refusing update from remote peer 7baca47cd2b04cbf9a68e5f0f6305a6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:41:21.409934 14846 raft_consensus.cc:1261] T 2c3aae985f0a4178a1f84f9f66898697 P 37e1e5c5ca634982a0a258e89333bf26 [term 1 FOLLOWER]: Refusing update from remote peer 7baca47cd2b04cbf9a68e5f0f6305a6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:41:21.410684 15180 consensus_queue.cc:1035] T 2c3aae985f0a4178a1f84f9f66898697 P 7baca47cd2b04cbf9a68e5f0f6305a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f52a7d693ec54ea19c27916605a799b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:41:21.410895 15183 consensus_queue.cc:1035] T 2c3aae985f0a4178a1f84f9f66898697 P 7baca47cd2b04cbf9a68e5f0f6305a6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "37e1e5c5ca634982a0a258e89333bf26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:41:21.418429 15188 mvcc.cc:205] Tried to move back new op lower bound from 7333793306248421376 to 7333793306179203072. Current Snapshot: MvccSnapshot[applied={T|T < 7333793306248421376}]
I20260927 02:41:21.425904 14703 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:48682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:41:21.426084 14703 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:48682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:41:21.428424 14703 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997: Sending DeleteTablet for 3 replicas of tablet 2c3aae985f0a4178a1f84f9f66898697
I20260927 02:41:21.429288 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 14742
I20260927 02:41:21.429347 14964 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c3aae985f0a4178a1f84f9f66898697 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:41:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60674
I20260927 02:41:21.429342 15096 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c3aae985f0a4178a1f84f9f66898697 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:41:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59312
I20260927 02:41:21.430078 15197 tablet_replica.cc:330] T 2c3aae985f0a4178a1f84f9f66898697 P 7baca47cd2b04cbf9a68e5f0f6305a6d: stopping tablet replica
I20260927 02:41:21.430423 15197 raft_consensus.cc:2229] T 2c3aae985f0a4178a1f84f9f66898697 P 7baca47cd2b04cbf9a68e5f0f6305a6d [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:41:21.430828 15197 raft_consensus.cc:2258] T 2c3aae985f0a4178a1f84f9f66898697 P 7baca47cd2b04cbf9a68e5f0f6305a6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:41:21.431193 15196 tablet_replica.cc:330] T 2c3aae985f0a4178a1f84f9f66898697 P f52a7d693ec54ea19c27916605a799b4: stopping tablet replica
I20260927 02:41:21.431499 15196 raft_consensus.cc:2229] T 2c3aae985f0a4178a1f84f9f66898697 P f52a7d693ec54ea19c27916605a799b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:41:21.431788 15196 raft_consensus.cc:2258] T 2c3aae985f0a4178a1f84f9f66898697 P f52a7d693ec54ea19c27916605a799b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:41:21.432605 15197 ts_tablet_manager.cc:1905] T 2c3aae985f0a4178a1f84f9f66898697 P 7baca47cd2b04cbf9a68e5f0f6305a6d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 02:41:21.435258 14689 connection.cc:535] client connection to 127.0.98.1:42077 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 02:41:21.435336 15197 ts_tablet_manager.cc:1918] T 2c3aae985f0a4178a1f84f9f66898697 P 7baca47cd2b04cbf9a68e5f0f6305a6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260927 02:41:21.435381 14689 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 02:41:21.435422 15197 log.cc:1196] T 2c3aae985f0a4178a1f84f9f66898697 P 7baca47cd2b04cbf9a68e5f0f6305a6d: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-2/wal/wals/2c3aae985f0a4178a1f84f9f66898697
I20260927 02:41:21.435736 15197 ts_tablet_manager.cc:1939] T 2c3aae985f0a4178a1f84f9f66898697 P 7baca47cd2b04cbf9a68e5f0f6305a6d: Deleting consensus metadata
I20260927 02:41:21.436102 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 14895
I20260927 02:41:21.438414 14688 catalog_manager.cc:4841] TS 7baca47cd2b04cbf9a68e5f0f6305a6d (127.0.98.3:44647): tablet 2c3aae985f0a4178a1f84f9f66898697 (table test-table [id=c36cc53479f34fb380f01a7a69f53a00]) successfully deleted
I20260927 02:41:21.438985 15172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:21.438693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42077 (local address 127.0.0.1:48262)
0927 02:41:21.438831 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:21.438904 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 02:41:21.439147 14689 catalog_manager.cc:4584] TS 37e1e5c5ca634982a0a258e89333bf26 (127.0.98.1:42077): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2c3aae985f0a4178a1f84f9f66898697: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42077: connect: Connection refused (error 111)
I20260927 02:41:21.441514 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 15033
I20260927 02:41:21.443061 15172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:21.441672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36519 (local address 127.0.0.1:60688)
0927 02:41:21.442955 (+ 1283us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:21.443002 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:36519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1216}
I20260927 02:41:21.446370 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 14671
I20260927 02:41:21.451375 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45947
--webserver_interface=127.0.98.62
--webserver_port=32953
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45947
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:41:21.558082 15199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:41:21.558374 15199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:41:21.558429 15199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:41:21.562433 15199 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:41:21.562496 15199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:41:21.562521 15199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:41:21.562541 15199 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:41:21.562561 15199 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:41:21.567747 15199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=32953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:41:21.568753 15199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:41:21.569743 15199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:41:21.577565 15207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:41:21.577620 15204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:41:21.577677 15199 server_base.cc:1034] running on GCE node
W20260927 02:41:21.577571 15205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:41:21.578382 15199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:41:21.579376 15199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:41:21.580539 15199 hybrid_clock.cc:648] HybridClock initialized: now 1790476881580511 us; error 38 us; skew 500 ppm
Sep 27 02:41:21 dist-test-slave-zv8z krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:41:21.583202 15199 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:41:21.584049 15199 webserver.cc:460] Webserver started at http://127.0.98.62:32953/ using document root <none> and password file <none>
I20260927 02:41:21.584568 15199 fs_manager.cc:359] Metadata directory not provided
I20260927 02:41:21.584632 15199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:41:21.588572 15199 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:41:21.590734 15214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:21.591979 15199 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:41:21.592092 15199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "d7409a2403734592bcce742551f20997"
format_stamp: "Formatted at 2026-09-27 02:41:19 on dist-test-slave-zv8z"
server_key: "32528d19bb181df7aa255d8c2a4e30f8"
server_key_iv: "b0beda8656ece766f07094794636c357"
server_key_version: "encryptionkey@0"
I20260927 02:41:21.592624 15199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:41:21.613960 15199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:41:21.617607 15199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:41:21.617751 15199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:41:21.624114 15199 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45947
I20260927 02:41:21.624364 15266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45947 every 8 connection(s)
I20260927 02:41:21.625120 15199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:41:21.626832 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 15199
I20260927 02:41:21.627202 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42077
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45947
--webserver_port=41129
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:41:21.629731 15267 sys_catalog.cc:263] Verifying existing consensus state
I20260927 02:41:21.631476 15267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997: Bootstrap starting.
I20260927 02:41:21.648370 15267 log.cc:824] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997: Log is configured to *not* fsync() on all Append() calls
I20260927 02:41:21.661119 15267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:41:21.661478 15267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997: Bootstrap complete.
I20260927 02:41:21.665621 15267 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7409a2403734592bcce742551f20997" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45947 } }
I20260927 02:41:21.665995 15267 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7409a2403734592bcce742551f20997, State: Initialized, Role: FOLLOWER
I20260927 02:41:21.666504 15267 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7409a2403734592bcce742551f20997" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45947 } }
I20260927 02:41:21.666620 15267 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:41:21.666679 15267 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:41:21.666774 15267 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997 [term 2 FOLLOWER]: Advancing to term 3
I20260927 02:41:21.668533 15267 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7409a2403734592bcce742551f20997" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45947 } }
I20260927 02:41:21.668830 15267 leader_election.cc:302] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7409a2403734592bcce742551f20997; no voters:
I20260927 02:41:21.669103 15267 leader_election.cc:288] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 02:41:21.669199 15271 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 02:41:21.669389 15271 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997 [term 3 LEADER]: Becoming Leader. State: Replica: d7409a2403734592bcce742551f20997, State: Running, Role: LEADER
I20260927 02:41:21.669673 15271 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7409a2403734592bcce742551f20997" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45947 } }
I20260927 02:41:21.670352 15267 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:41:21.670691 15273 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7409a2403734592bcce742551f20997. Latest consensus state: current_term: 3 leader_uuid: "d7409a2403734592bcce742551f20997" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7409a2403734592bcce742551f20997" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45947 } } }
I20260927 02:41:21.670928 15273 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997 [sys.catalog]: This master's current role is: LEADER
I20260927 02:41:21.671115 15272 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d7409a2403734592bcce742551f20997" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7409a2403734592bcce742551f20997" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45947 } } }
I20260927 02:41:21.671221 15272 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7409a2403734592bcce742551f20997 [sys.catalog]: This master's current role is: LEADER
I20260927 02:41:21.671518 15276 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:41:21.676568 15276 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:41:21.677387 15276 catalog_manager.cc:1259] Loaded cluster ID: 113bd70eda5b40158c169ffd208d3f94
I20260927 02:41:21.677441 15276 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 02:41:21.678396 15276 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fee482fb9ad google::LogMessage::Fail() at ??:0
@ 0x7fee482fe88a google::LogMessage::SendToLog() at ??:0
@ 0x7fee482fb4de google::LogMessage::Flush() at ??:0
@ 0x7fee482fca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fee52c3afdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fee52c3bd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fee52c38579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fee52c7c640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fee4a54d3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fee4a54dc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fee4a54f3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fee4a53e512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fee4bb826db start_thread at ??:0
@ 0x7fee46df961f clone at ??:0
W20260927 02:41:21.752300 15269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:41:21.752599 15269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:41:21.752647 15269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:41:21.756922 15269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:41:21.756983 15269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:41:21.757059 15269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:41:21.762387 15269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42077
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41129
--tserver_master_addrs=127.0.98.62:45947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:41:21.763412 15269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:41:21.764374 15269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:41:21.773586 15308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:41:21.773741 15269 server_base.cc:1034] running on GCE node
W20260927 02:41:21.773586 15309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:41:21.773581 15311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:41:21.774399 15269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:41:21.774969 15269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:41:21.776124 15269 hybrid_clock.cc:648] HybridClock initialized: now 1790476881776096 us; error 46 us; skew 500 ppm
Sep 27 02:41:21 dist-test-slave-zv8z krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:41:21.778692 15269 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:41:21.779510 15269 webserver.cc:460] Webserver started at http://127.0.98.1:41129/ using document root <none> and password file <none>
I20260927 02:41:21.780025 15269 fs_manager.cc:359] Metadata directory not provided
I20260927 02:41:21.780081 15269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:41:21.783766 15269 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:41:21.785887 15318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:21.787195 15269 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:41:21.787298 15269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-0/wal
uuid: "37e1e5c5ca634982a0a258e89333bf26"
format_stamp: "Formatted at 2026-09-27 02:41:19 on dist-test-slave-zv8z"
server_key: "f6671aefcba165e019246ed6a0575046"
server_key_iv: "80b76de96e47d39b5ce1ae98f3f8539f"
server_key_version: "encryptionkey@0"
I20260927 02:41:21.787801 15269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:41:21.804261 15269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:41:21.807365 15269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:41:21.807538 15269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:41:21.808175 15269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:41:21.809248 15325 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 02:41:21.814443 15269 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 02:41:21.814500 15269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260927 02:41:21.814563 15269 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 02:41:21.816138 15269 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 02:41:21.816179 15269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 02:41:21.816509 15325 tablet_bootstrap.cc:492] T 2c3aae985f0a4178a1f84f9f66898697 P 37e1e5c5ca634982a0a258e89333bf26: Bootstrap starting.
I20260927 02:41:21.825538 15269 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42077
I20260927 02:41:21.825816 15432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42077 every 8 connection(s)
I20260927 02:41:21.826567 15269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:41:21.833830 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 15269
I20260927 02:41:21.834225 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36519
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45947
--webserver_port=44993
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:41:21.837844 15325 log.cc:824] T 2c3aae985f0a4178a1f84f9f66898697 P 37e1e5c5ca634982a0a258e89333bf26: Log is configured to *not* fsync() on all Append() calls
I20260927 02:41:21.846477 15325 tablet_bootstrap.cc:492] T 2c3aae985f0a4178a1f84f9f66898697 P 37e1e5c5ca634982a0a258e89333bf26: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:41:21.846915 15325 tablet_bootstrap.cc:492] T 2c3aae985f0a4178a1f84f9f66898697 P 37e1e5c5ca634982a0a258e89333bf26: Bootstrap complete.
I20260927 02:41:21.848209 15325 ts_tablet_manager.cc:1397] T 2c3aae985f0a4178a1f84f9f66898697 P 37e1e5c5ca634982a0a258e89333bf26: Time spent bootstrapping tablet: real 0.032s user 0.022s sys 0.008s
I20260927 02:41:21.851270 15325 raft_consensus.cc:348] T 2c3aae985f0a4178a1f84f9f66898697 P 37e1e5c5ca634982a0a258e89333bf26 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7baca47cd2b04cbf9a68e5f0f6305a6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44647 } } peers { permanent_uuid: "f52a7d693ec54ea19c27916605a799b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36519 } } peers { permanent_uuid: "37e1e5c5ca634982a0a258e89333bf26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42077 } }
I20260927 02:41:21.851801 15325 raft_consensus.cc:729] T 2c3aae985f0a4178a1f84f9f66898697 P 37e1e5c5ca634982a0a258e89333bf26 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37e1e5c5ca634982a0a258e89333bf26, State: Initialized, Role: FOLLOWER
I20260927 02:41:21.852331 15325 consensus_queue.cc:260] T 2c3aae985f0a4178a1f84f9f66898697 P 37e1e5c5ca634982a0a258e89333bf26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7baca47cd2b04cbf9a68e5f0f6305a6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44647 } } peers { permanent_uuid: "f52a7d693ec54ea19c27916605a799b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36519 } } peers { permanent_uuid: "37e1e5c5ca634982a0a258e89333bf26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42077 } }
I20260927 02:41:21.853360 15325 ts_tablet_manager.cc:1428] T 2c3aae985f0a4178a1f84f9f66898697 P 37e1e5c5ca634982a0a258e89333bf26: Time spent starting tablet: real 0.005s user 0.006s sys 0.001s
W20260927 02:41:21.902976 15435 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:41:21.828402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.1:48489)
0927 02:41:21.828870 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:21.828923 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 02:41:21.829735 (+ 812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:41:21.902215 (+ 72480us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":121,"threads_started":1}
W20260927 02:41:21.903540 15433 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45947 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45947: Client connection negotiation failed: client connection to 127.0.98.62:45947: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 02:41:21.904312 15435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:21.904018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.1:60839)
0927 02:41:21.904177 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:21.904253 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 02:41:21.905035 15435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:21.904795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.1:58541)
0927 02:41:21.904919 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:21.904973 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 02:41:21.905251 15433 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 02:41:21.943351 15436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:41:21.943573 15436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:41:21.943612 15436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:41:21.947453 15436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:41:21.947499 15436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:41:21.947552 15436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:41:21.952590 15436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36519
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44993
--tserver_master_addrs=127.0.98.62:45947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:41:21.953583 15436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:41:21.954602 15436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:41:21.963845 15436 server_base.cc:1034] running on GCE node
W20260927 02:41:21.963860 15443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:41:21.963800 15444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:41:21.963824 15446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:41:21.964377 15436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:41:21.964984 15436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:41:21.966140 15436 hybrid_clock.cc:648] HybridClock initialized: now 1790476881966128 us; error 38 us; skew 500 ppm
Sep 27 02:41:21 dist-test-slave-zv8z krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:41:21.968762 15436 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:41:21.969586 15436 webserver.cc:460] Webserver started at http://127.0.98.2:44993/ using document root <none> and password file <none>
I20260927 02:41:21.970084 15436 fs_manager.cc:359] Metadata directory not provided
I20260927 02:41:21.970146 15436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:41:21.973766 15436 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:41:21.975782 15453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:21.976998 15436 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:41:21.977097 15436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-1/wal
uuid: "f52a7d693ec54ea19c27916605a799b4"
format_stamp: "Formatted at 2026-09-27 02:41:19 on dist-test-slave-zv8z"
server_key: "a0c4d48df42b09f807c9e078eb24d97a"
server_key_iv: "2e69127122017f700a318d37feb1df45"
server_key_version: "encryptionkey@0"
I20260927 02:41:21.977607 15436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:41:21.988013 15436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:41:21.990794 15436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:41:21.990954 15436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:41:21.991529 15436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:41:21.992580 15460 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 02:41:21.996683 15436 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 02:41:21.996744 15436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260927 02:41:21.996804 15436 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 02:41:21.998449 15436 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 02:41:21.998492 15436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260927 02:41:21.998744 15460 tablet_bootstrap.cc:492] T 2c3aae985f0a4178a1f84f9f66898697 P f52a7d693ec54ea19c27916605a799b4: Bootstrap starting.
I20260927 02:41:22.008227 15436 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36519
I20260927 02:41:22.008329 15567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36519 every 8 connection(s)
I20260927 02:41:22.009147 15436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:41:22.010205 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 15436
I20260927 02:41:22.010731 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44647
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45947
--webserver_port=40909
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:41:22.012501 15570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:22.011046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.2:57521)
0927 02:41:22.011515 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:22.011638 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":117,"threads_started":1}
W20260927 02:41:22.012934 15568 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45947 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45947: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
I20260927 02:41:22.014595 15570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:22.013424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.2:59189)
0927 02:41:22.014462 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:22.014521 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":961,"mutex_wait_us":910}
I20260927 02:41:22.015151 15570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:22.014979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.2:49737)
0927 02:41:22.015056 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:22.015102 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260927 02:41:22.015300 15568 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 02:41:22.016752 15460 log.cc:824] T 2c3aae985f0a4178a1f84f9f66898697 P f52a7d693ec54ea19c27916605a799b4: Log is configured to *not* fsync() on all Append() calls
I20260927 02:41:22.023298 15460 tablet_bootstrap.cc:492] T 2c3aae985f0a4178a1f84f9f66898697 P f52a7d693ec54ea19c27916605a799b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:41:22.023592 15460 tablet_bootstrap.cc:492] T 2c3aae985f0a4178a1f84f9f66898697 P f52a7d693ec54ea19c27916605a799b4: Bootstrap complete.
I20260927 02:41:22.024484 15460 ts_tablet_manager.cc:1397] T 2c3aae985f0a4178a1f84f9f66898697 P f52a7d693ec54ea19c27916605a799b4: Time spent bootstrapping tablet: real 0.026s user 0.014s sys 0.009s
I20260927 02:41:22.027163 15460 raft_consensus.cc:348] T 2c3aae985f0a4178a1f84f9f66898697 P f52a7d693ec54ea19c27916605a799b4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7baca47cd2b04cbf9a68e5f0f6305a6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44647 } } peers { permanent_uuid: "f52a7d693ec54ea19c27916605a799b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36519 } } peers { permanent_uuid: "37e1e5c5ca634982a0a258e89333bf26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42077 } }
I20260927 02:41:22.027580 15460 raft_consensus.cc:729] T 2c3aae985f0a4178a1f84f9f66898697 P f52a7d693ec54ea19c27916605a799b4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f52a7d693ec54ea19c27916605a799b4, State: Initialized, Role: FOLLOWER
I20260927 02:41:22.027995 15460 consensus_queue.cc:260] T 2c3aae985f0a4178a1f84f9f66898697 P f52a7d693ec54ea19c27916605a799b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7baca47cd2b04cbf9a68e5f0f6305a6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44647 } } peers { permanent_uuid: "f52a7d693ec54ea19c27916605a799b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36519 } } peers { permanent_uuid: "37e1e5c5ca634982a0a258e89333bf26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42077 } }
I20260927 02:41:22.028942 15460 ts_tablet_manager.cc:1428] T 2c3aae985f0a4178a1f84f9f66898697 P f52a7d693ec54ea19c27916605a799b4: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20260927 02:41:22.029274 15570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:22.029010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.2:60065)
0927 02:41:22.029137 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:22.029198 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 02:41:22.029881 15570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:22.029674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.2:48155)
0927 02:41:22.029769 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:22.029819 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 02:41:22.122040 15571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:41:22.122334 15571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:41:22.122381 15571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:41:22.126273 15571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:41:22.126319 15571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:41:22.126379 15571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:41:22.131465 15571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44647
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40909
--tserver_master_addrs=127.0.98.62:45947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:41:22.132512 15571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:41:22.133512 15571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:41:22.142979 15581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:41:22.142974 15579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:41:22.143025 15571 server_base.cc:1034] running on GCE node
W20260927 02:41:22.142974 15578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:41:22.143666 15571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:41:22.144315 15571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:41:22.145483 15571 hybrid_clock.cc:648] HybridClock initialized: now 1790476882145464 us; error 34 us; skew 500 ppm
Sep 27 02:41:22 dist-test-slave-zv8z krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476882, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:41:22.148237 15571 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:41:22.149183 15571 webserver.cc:460] Webserver started at http://127.0.98.3:40909/ using document root <none> and password file <none>
I20260927 02:41:22.149730 15571 fs_manager.cc:359] Metadata directory not provided
I20260927 02:41:22.149799 15571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:41:22.153869 15571 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 02:41:22.156291 15588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:22.157792 15571 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 02:41:22.157923 15571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-2/wal
uuid: "7baca47cd2b04cbf9a68e5f0f6305a6d"
format_stamp: "Formatted at 2026-09-27 02:41:20 on dist-test-slave-zv8z"
server_key: "bfdc1c2c2d5f6176ee1e7fc64974b164"
server_key_iv: "9f41d3d33bab1866beaa4909769508f3"
server_key_version: "encryptionkey@0"
I20260927 02:41:22.158548 15571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:41:22.177448 15571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:41:22.180560 15571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:41:22.180739 15571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:41:22.181349 15571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:41:22.182132 15571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:41:22.182178 15571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:22.182230 15571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:41:22.182278 15571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:22.190634 15571 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44647
I20260927 02:41:22.190670 15701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44647 every 8 connection(s)
I20260927 02:41:22.191730 15571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:41:22.195782 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:22.193929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.3:36415)
0927 02:41:22.194475 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:22.194976 (+ 501us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":130,"threads_started":1}
W20260927 02:41:22.196242 15702 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45947 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45947: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
I20260927 02:41:22.196894 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:22.196619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.3:38037)
0927 02:41:22.196779 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:22.196839 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 02:41:22.197458 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:22.197286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.3:38823)
0927 02:41:22.197371 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:22.197411 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 02:41:22.197613 15702 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 02:41:22.199633 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 15571
I20260927 02:41:22.906291 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:22.905820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.1:35947)
0927 02:41:22.906088 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:22.906148 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":68,"threads_started":1}
I20260927 02:41:23.030898 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:23.030516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.2:36991)
0927 02:41:23.030745 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:23.030801 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":67,"threads_started":1}
I20260927 02:41:23.198654 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:23.198208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.3:50641)
0927 02:41:23.198486 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:23.198546 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":97,"threads_started":1}
I20260927 02:41:23.215266 15709 raft_consensus.cc:482] T 2c3aae985f0a4178a1f84f9f66898697 P 37e1e5c5ca634982a0a258e89333bf26 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:41:23.215524 15709 raft_consensus.cc:504] T 2c3aae985f0a4178a1f84f9f66898697 P 37e1e5c5ca634982a0a258e89333bf26 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7baca47cd2b04cbf9a68e5f0f6305a6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44647 } } peers { permanent_uuid: "f52a7d693ec54ea19c27916605a799b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36519 } } peers { permanent_uuid: "37e1e5c5ca634982a0a258e89333bf26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42077 } }
I20260927 02:41:23.216603 15709 leader_election.cc:288] T 2c3aae985f0a4178a1f84f9f66898697 P 37e1e5c5ca634982a0a258e89333bf26 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7baca47cd2b04cbf9a68e5f0f6305a6d (127.0.98.3:44647), f52a7d693ec54ea19c27916605a799b4 (127.0.98.2:36519)
Sep 27 02:41:23 dist-test-slave-zv8z krb5kdc[14122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 27 02:41:23 dist-test-slave-zv8z krb5kdc[14122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260927 02:41:23.229346 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:23.216846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44647 (local address 127.0.98.1:50539)
0927 02:41:23.217123 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:23.217145 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 02:41:23.217287 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:41:23.218714 (+ 1427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:41:23.218717 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:41:23.218970 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:41:23.219336 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:23.219353 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:23.220552 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:23.220555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:23.221467 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:23.221477 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:23.222161 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:23.222165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:23.222437 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:23.223059 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:41:23.223075 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:41:23.224796 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:41:23.227092 (+ 2296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:23.227095 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:23.227097 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:23.227412 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:23.227925 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:23.227928 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:23.227930 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:23.228001 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:23.228527 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:41:23.228530 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:41:23.228656 (+ 126us) client_negotiation.cc:769] Sending connection context
0927 02:41:23.228814 (+ 158us) client_negotiation.cc:240] Negotiation successful
0927 02:41:23.229151 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":186,"spinlock_wait_cycles":17280,"thread_start_us":77,"threads_started":1}
I20260927 02:41:23.229357 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:23.217024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50539 (local address 127.0.98.3:44647)
0927 02:41:23.217421 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:23.217450 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:23.217486 (+ 36us) server_negotiation.cc:407] Connection header received
0927 02:41:23.218145 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:23.218150 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:23.218496 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:23.218742 (+ 246us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:41:23.219508 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:23.220435 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:23.221580 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:23.222052 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:23.222190 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:23.224996 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:41:23.225002 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:41:23.225016 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:41:23.225043 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:41:23.226957 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:23.227735 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:23.227739 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:23.227743 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:23.227839 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:23.228078 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:23.228081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:23.228082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:23.228429 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:41:23.228538 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:23.228932 (+ 394us) server_negotiation.cc:299] Negotiation successful
0927 02:41:23.229178 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":105,"threads_started":1}
I20260927 02:41:23.229382 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:23.217037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53177 (local address 127.0.98.2:36519)
0927 02:41:23.217488 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:23.217498 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:23.217529 (+ 31us) server_negotiation.cc:407] Connection header received
0927 02:41:23.217717 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:23.217721 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:23.218048 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:23.218353 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:41:23.219495 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:23.220423 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:23.221578 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:23.222042 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:23.222188 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:23.224681 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:41:23.224686 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:41:23.224699 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:41:23.224725 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:41:23.226955 (+ 2230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:23.227513 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:23.227519 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:23.227524 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:23.227631 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:23.227911 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:23.227914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:23.227916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:23.228312 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:41:23.228429 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:23.228945 (+ 516us) server_negotiation.cc:299] Negotiation successful
0927 02:41:23.229200 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":86,"threads_started":1}
I20260927 02:41:23.229380 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:23.216846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36519 (local address 127.0.98.1:53177)
0927 02:41:23.217123 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:23.217139 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 02:41:23.217253 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:41:23.218416 (+ 1163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:41:23.218421 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:41:23.218886 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:41:23.219336 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:23.219351 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:23.220557 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:23.220559 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:23.221469 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:23.221478 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:23.222155 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:23.222159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:23.222434 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:23.223108 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:41:23.223120 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:41:23.224497 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:41:23.227078 (+ 2581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:23.227082 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:23.227097 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:23.227407 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:23.227702 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:23.227704 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:23.227705 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:23.227819 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:23.228422 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:41:23.228429 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:41:23.228667 (+ 238us) client_negotiation.cc:769] Sending connection context
0927 02:41:23.228814 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 02:41:23.229223 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":267,"spinlock_wait_cycles":8448}
I20260927 02:41:23.230785 15522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3aae985f0a4178a1f84f9f66898697" candidate_uuid: "37e1e5c5ca634982a0a258e89333bf26" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f52a7d693ec54ea19c27916605a799b4" is_pre_election: true
I20260927 02:41:23.231065 15522 raft_consensus.cc:2454] T 2c3aae985f0a4178a1f84f9f66898697 P f52a7d693ec54ea19c27916605a799b4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37e1e5c5ca634982a0a258e89333bf26 in term 1.
W20260927 02:41:23.231894 15319 leader_election.cc:341] T 2c3aae985f0a4178a1f84f9f66898697 P 37e1e5c5ca634982a0a258e89333bf26 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7baca47cd2b04cbf9a68e5f0f6305a6d (127.0.98.3:44647): Not found: Tablet not found: 2c3aae985f0a4178a1f84f9f66898697
I20260927 02:41:23.232024 15320 leader_election.cc:302] T 2c3aae985f0a4178a1f84f9f66898697 P 37e1e5c5ca634982a0a258e89333bf26 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37e1e5c5ca634982a0a258e89333bf26, f52a7d693ec54ea19c27916605a799b4; no voters: 7baca47cd2b04cbf9a68e5f0f6305a6d
I20260927 02:41:23.232270 15709 raft_consensus.cc:2781] T 2c3aae985f0a4178a1f84f9f66898697 P 37e1e5c5ca634982a0a258e89333bf26 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 02:41:23.232345 15709 raft_consensus.cc:482] T 2c3aae985f0a4178a1f84f9f66898697 P 37e1e5c5ca634982a0a258e89333bf26 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:41:23.232398 15709 raft_consensus.cc:3037] T 2c3aae985f0a4178a1f84f9f66898697 P 37e1e5c5ca634982a0a258e89333bf26 [term 1 FOLLOWER]: Advancing to term 2
I20260927 02:41:23.234512 15709 raft_consensus.cc:504] T 2c3aae985f0a4178a1f84f9f66898697 P 37e1e5c5ca634982a0a258e89333bf26 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7baca47cd2b04cbf9a68e5f0f6305a6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44647 } } peers { permanent_uuid: "f52a7d693ec54ea19c27916605a799b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36519 } } peers { permanent_uuid: "37e1e5c5ca634982a0a258e89333bf26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42077 } }
I20260927 02:41:23.234895 15709 leader_election.cc:288] T 2c3aae985f0a4178a1f84f9f66898697 P 37e1e5c5ca634982a0a258e89333bf26 [CANDIDATE]: Term 2 election: Requested vote from peers 7baca47cd2b04cbf9a68e5f0f6305a6d (127.0.98.3:44647), f52a7d693ec54ea19c27916605a799b4 (127.0.98.2:36519)
I20260927 02:41:23.235384 15522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c3aae985f0a4178a1f84f9f66898697" candidate_uuid: "37e1e5c5ca634982a0a258e89333bf26" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f52a7d693ec54ea19c27916605a799b4"
I20260927 02:41:23.235519 15522 raft_consensus.cc:3037] T 2c3aae985f0a4178a1f84f9f66898697 P f52a7d693ec54ea19c27916605a799b4 [term 1 FOLLOWER]: Advancing to term 2
W20260927 02:41:23.235706 15319 leader_election.cc:341] T 2c3aae985f0a4178a1f84f9f66898697 P 37e1e5c5ca634982a0a258e89333bf26 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 7baca47cd2b04cbf9a68e5f0f6305a6d (127.0.98.3:44647): Not found: Tablet not found: 2c3aae985f0a4178a1f84f9f66898697
I20260927 02:41:23.237789 15522 raft_consensus.cc:2454] T 2c3aae985f0a4178a1f84f9f66898697 P f52a7d693ec54ea19c27916605a799b4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37e1e5c5ca634982a0a258e89333bf26 in term 2.
I20260927 02:41:23.238137 15320 leader_election.cc:302] T 2c3aae985f0a4178a1f84f9f66898697 P 37e1e5c5ca634982a0a258e89333bf26 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37e1e5c5ca634982a0a258e89333bf26, f52a7d693ec54ea19c27916605a799b4; no voters: 7baca47cd2b04cbf9a68e5f0f6305a6d
I20260927 02:41:23.238417 15709 raft_consensus.cc:2781] T 2c3aae985f0a4178a1f84f9f66898697 P 37e1e5c5ca634982a0a258e89333bf26 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 02:41:23.238583 15709 raft_consensus.cc:686] T 2c3aae985f0a4178a1f84f9f66898697 P 37e1e5c5ca634982a0a258e89333bf26 [term 2 LEADER]: Becoming Leader. State: Replica: 37e1e5c5ca634982a0a258e89333bf26, State: Running, Role: LEADER
I20260927 02:41:23.238874 15709 consensus_queue.cc:237] T 2c3aae985f0a4178a1f84f9f66898697 P 37e1e5c5ca634982a0a258e89333bf26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7baca47cd2b04cbf9a68e5f0f6305a6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44647 } } peers { permanent_uuid: "f52a7d693ec54ea19c27916605a799b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36519 } } peers { permanent_uuid: "37e1e5c5ca634982a0a258e89333bf26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42077 } }
I20260927 02:41:23.239276 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:23.239047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.1:51437)
0927 02:41:23.239152 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:23.239204 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 02:41:23.662154 15319 consensus_peers.cc:487] T 2c3aae985f0a4178a1f84f9f66898697 P 37e1e5c5ca634982a0a258e89333bf26 -> Peer 7baca47cd2b04cbf9a68e5f0f6305a6d (127.0.98.3:44647): Couldn't send request to peer 7baca47cd2b04cbf9a68e5f0f6305a6d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2c3aae985f0a4178a1f84f9f66898697. This is attempt 1: this message will repeat every 5th retry.
I20260927 02:41:23.811519 15522 raft_consensus.cc:1261] T 2c3aae985f0a4178a1f84f9f66898697 P f52a7d693ec54ea19c27916605a799b4 [term 2 FOLLOWER]: Refusing update from remote peer 37e1e5c5ca634982a0a258e89333bf26: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260927 02:41:23.812232 15709 consensus_queue.cc:1035] T 2c3aae985f0a4178a1f84f9f66898697 P 37e1e5c5ca634982a0a258e89333bf26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f52a7d693ec54ea19c27916605a799b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 02:41:23.814225 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:23.813895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.2:33303)
0927 02:41:23.814084 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:23.814140 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":54,"threads_started":1}
I20260927 02:41:23.816635 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:23.816221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.1:51975)
0927 02:41:23.816459 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:23.816517 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":49,"threads_started":1}
I20260927 02:41:23.816695 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:23.816441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.2:37259)
0927 02:41:23.816588 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:23.816636 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 02:41:24.122498 15722 ts_tablet_manager.cc:927] T 2c3aae985f0a4178a1f84f9f66898697 P 7baca47cd2b04cbf9a68e5f0f6305a6d: Initiating tablet copy from peer 37e1e5c5ca634982a0a258e89333bf26 (127.0.98.1:42077)
I20260927 02:41:24.123055 15722 tablet_copy_client.cc:316] T 2c3aae985f0a4178a1f84f9f66898697 P 7baca47cd2b04cbf9a68e5f0f6305a6d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.1:42077
Sep 27 02:41:24 dist-test-slave-zv8z krb5kdc[14122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476882, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:41:24.132789 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:24.123560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42077 (local address 127.0.98.3:45229)
0927 02:41:24.123881 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:24.123901 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 02:41:24.124023 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:41:24.124516 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:41:24.124521 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:41:24.124947 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:41:24.125065 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:24.125072 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:24.126069 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:24.126072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:24.126791 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:24.126801 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:24.127387 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:24.127390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:24.127484 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:24.128000 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:41:24.128014 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:41:24.129381 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:41:24.131133 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:24.131143 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:24.131150 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:24.131373 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:24.131647 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:24.131650 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:24.131652 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:24.131727 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:24.132157 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:41:24.132164 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:41:24.132283 (+ 119us) client_negotiation.cc:769] Sending connection context
0927 02:41:24.132407 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 02:41:24.132577 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":95,"threads_started":1}
I20260927 02:41:24.132800 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:24.123729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45229 (local address 127.0.98.1:42077)
0927 02:41:24.124094 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:24.124104 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:24.124124 (+ 20us) server_negotiation.cc:407] Connection header received
0927 02:41:24.124169 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:24.124174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:24.124349 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:24.124473 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:41:24.125197 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:24.125949 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:24.126918 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:24.127288 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:24.127373 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:24.129521 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:41:24.129532 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:41:24.129548 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:41:24.129573 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:41:24.131028 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:24.131482 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:24.131487 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:24.131489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:24.131564 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:24.131803 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:24.131806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:24.131808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:24.132049 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:41:24.132167 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:24.132513 (+ 346us) server_negotiation.cc:299] Negotiation successful
0927 02:41:24.132657 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":87,"threads_started":1}
I20260927 02:41:24.133950 15407 tablet_copy_service.cc:140] P 37e1e5c5ca634982a0a258e89333bf26: Received BeginTabletCopySession request for tablet 2c3aae985f0a4178a1f84f9f66898697 from peer 7baca47cd2b04cbf9a68e5f0f6305a6d ({username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45229)
I20260927 02:41:24.134127 15407 tablet_copy_service.cc:161] P 37e1e5c5ca634982a0a258e89333bf26: Beginning new tablet copy session on tablet 2c3aae985f0a4178a1f84f9f66898697 from peer 7baca47cd2b04cbf9a68e5f0f6305a6d at {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45229: session id = 7baca47cd2b04cbf9a68e5f0f6305a6d-2c3aae985f0a4178a1f84f9f66898697
I20260927 02:41:24.135300 15407 tablet_copy_source_session.cc:205] T 2c3aae985f0a4178a1f84f9f66898697 P 37e1e5c5ca634982a0a258e89333bf26: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 02:41:24.139246 15722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c3aae985f0a4178a1f84f9f66898697. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:41:24.145717 15722 tablet_copy_client.cc:791] T 2c3aae985f0a4178a1f84f9f66898697 P 7baca47cd2b04cbf9a68e5f0f6305a6d: tablet copy: Starting download of 0 data blocks...
I20260927 02:41:24.145895 15722 tablet_copy_client.cc:655] T 2c3aae985f0a4178a1f84f9f66898697 P 7baca47cd2b04cbf9a68e5f0f6305a6d: tablet copy: Starting download of 1 WAL segments...
I20260927 02:41:24.147728 15722 tablet_copy_client.cc:531] T 2c3aae985f0a4178a1f84f9f66898697 P 7baca47cd2b04cbf9a68e5f0f6305a6d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 02:41:24.149446 15722 tablet_bootstrap.cc:492] T 2c3aae985f0a4178a1f84f9f66898697 P 7baca47cd2b04cbf9a68e5f0f6305a6d: Bootstrap starting.
I20260927 02:41:24.154886 15722 log.cc:824] T 2c3aae985f0a4178a1f84f9f66898697 P 7baca47cd2b04cbf9a68e5f0f6305a6d: Log is configured to *not* fsync() on all Append() calls
I20260927 02:41:24.161866 15722 tablet_bootstrap.cc:492] T 2c3aae985f0a4178a1f84f9f66898697 P 7baca47cd2b04cbf9a68e5f0f6305a6d: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:41:24.162242 15722 tablet_bootstrap.cc:492] T 2c3aae985f0a4178a1f84f9f66898697 P 7baca47cd2b04cbf9a68e5f0f6305a6d: Bootstrap complete.
I20260927 02:41:24.162603 15722 ts_tablet_manager.cc:1397] T 2c3aae985f0a4178a1f84f9f66898697 P 7baca47cd2b04cbf9a68e5f0f6305a6d: Time spent bootstrapping tablet: real 0.013s user 0.014s sys 0.000s
I20260927 02:41:24.164889 15722 raft_consensus.cc:348] T 2c3aae985f0a4178a1f84f9f66898697 P 7baca47cd2b04cbf9a68e5f0f6305a6d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7baca47cd2b04cbf9a68e5f0f6305a6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44647 } } peers { permanent_uuid: "f52a7d693ec54ea19c27916605a799b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36519 } } peers { permanent_uuid: "37e1e5c5ca634982a0a258e89333bf26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42077 } }
I20260927 02:41:24.165140 15722 raft_consensus.cc:729] T 2c3aae985f0a4178a1f84f9f66898697 P 7baca47cd2b04cbf9a68e5f0f6305a6d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7baca47cd2b04cbf9a68e5f0f6305a6d, State: Initialized, Role: FOLLOWER
I20260927 02:41:24.165537 15722 consensus_queue.cc:260] T 2c3aae985f0a4178a1f84f9f66898697 P 7baca47cd2b04cbf9a68e5f0f6305a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7baca47cd2b04cbf9a68e5f0f6305a6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44647 } } peers { permanent_uuid: "f52a7d693ec54ea19c27916605a799b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36519 } } peers { permanent_uuid: "37e1e5c5ca634982a0a258e89333bf26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42077 } }
I20260927 02:41:24.166467 15722 ts_tablet_manager.cc:1428] T 2c3aae985f0a4178a1f84f9f66898697 P 7baca47cd2b04cbf9a68e5f0f6305a6d: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260927 02:41:24.166746 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:24.166497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.3:45691)
0927 02:41:24.166620 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:24.166675 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 02:41:24.167132 15407 tablet_copy_service.cc:342] P 37e1e5c5ca634982a0a258e89333bf26: Request end of tablet copy session 7baca47cd2b04cbf9a68e5f0f6305a6d-2c3aae985f0a4178a1f84f9f66898697 received from {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45229
I20260927 02:41:24.167237 15407 tablet_copy_service.cc:433] P 37e1e5c5ca634982a0a258e89333bf26: ending tablet copy session 7baca47cd2b04cbf9a68e5f0f6305a6d-2c3aae985f0a4178a1f84f9f66898697 on tablet 2c3aae985f0a4178a1f84f9f66898697 with peer 7baca47cd2b04cbf9a68e5f0f6305a6d
I20260927 02:41:24.167374 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:24.167151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.3:53867)
0927 02:41:24.167262 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:24.167311 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 02:41:24.623839 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:24.623499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.1:54043)
0927 02:41:24.623705 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:24.623756 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":47,"threads_started":1}
I20260927 02:41:24.623909 15656 raft_consensus.cc:1203] T 2c3aae985f0a4178a1f84f9f66898697 P 7baca47cd2b04cbf9a68e5f0f6305a6d [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260927 02:41:24.624815 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:24.624527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.3:51853)
0927 02:41:24.624646 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:24.624740 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 02:41:24.817700 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:24.817301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.2:47811)
0927 02:41:24.817549 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:24.817606 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":67,"threads_started":1}
I20260927 02:41:25.625010 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:25.624561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.1:50625)
0927 02:41:25.624846 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:25.624905 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":76,"threads_started":1}
I20260927 02:41:25.625839 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:25.625449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.3:46185)
0927 02:41:25.625683 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:25.625747 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":67,"threads_started":1}
I20260927 02:41:25.818974 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:25.818506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.2:59889)
0927 02:41:25.818819 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:25.818877 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":69,"threads_started":1}
I20260927 02:41:26.626148 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:26.625736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.1:32817)
0927 02:41:26.625972 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:26.626027 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":70,"threads_started":1}
I20260927 02:41:26.626922 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:26.626550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.3:48205)
0927 02:41:26.626799 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:26.626848 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":62,"threads_started":1}
I20260927 02:41:26.820127 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:26.819706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.2:37565)
0927 02:41:26.819977 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:26.820032 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":73,"threads_started":1}
I20260927 02:41:27.627633 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:27.627138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.1:58759)
0927 02:41:27.627447 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:27.627520 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":68,"threads_started":1}
I20260927 02:41:27.628402 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:27.628170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.1:59161)
0927 02:41:27.628302 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:27.628348 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 02:41:27.628422 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:27.627972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.3:36043)
0927 02:41:27.628292 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:27.628344 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":84,"threads_started":1}
I20260927 02:41:27.821554 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:27.821054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.2:54683)
0927 02:41:27.821406 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:27.821459 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":102,"threads_started":1}
I20260927 02:41:28.629557 15741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:28.629135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.1:52961)
0927 02:41:28.629407 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:28.629460 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":81,"threads_started":1}
I20260927 02:41:28.629598 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:28.629132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.3:51265)
0927 02:41:28.629452 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:28.629506 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":85,"threads_started":1}
I20260927 02:41:28.822749 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:28.822340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.2:42199)
0927 02:41:28.822591 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:28.822651 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":81,"threads_started":1}
I20260927 02:41:29.630811 15744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:29.630376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.1:35831)
0927 02:41:29.630651 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:29.630718 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":85,"threads_started":1}
I20260927 02:41:29.630864 15745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:29.630420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.3:34063)
0927 02:41:29.630718 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:29.630772 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":71,"threads_started":1}
I20260927 02:41:29.824105 15746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:29.823634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.2:49141)
0927 02:41:29.823934 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:29.824007 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":83,"threads_started":1}
I20260927 02:41:30.631987 15747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:30.631555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.1:53403)
0927 02:41:30.631825 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:30.631887 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":93,"threads_started":1}
I20260927 02:41:30.632025 15748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:30.631603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.3:52141)
0927 02:41:30.631883 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:30.631937 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":60,"threads_started":1}
I20260927 02:41:30.825325 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:30.824909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.2:56547)
0927 02:41:30.825167 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:30.825230 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":79,"threads_started":1}
I20260927 02:41:31.633252 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:31.632822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.3:34911)
0927 02:41:31.633097 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:31.633158 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":85,"threads_started":1}
I20260927 02:41:31.633365 15750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:31.632820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.1:45381)
0927 02:41:31.633160 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:31.633214 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":102,"threads_started":1}
I20260927 02:41:31.826611 15752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:31.826148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.2:57807)
0927 02:41:31.826448 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:31.826515 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":131,"threads_started":1}
I20260927 02:41:32.634624 15753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:32.634174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.1:50853)
0927 02:41:32.634480 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:32.634535 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":100,"threads_started":1}
I20260927 02:41:32.634625 15754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:32.634168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.3:45203)
0927 02:41:32.634483 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:32.634535 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":95,"threads_started":1}
I20260927 02:41:32.827891 15755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:32.827442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.2:53973)
0927 02:41:32.827712 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:32.827791 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":86,"threads_started":1}
I20260927 02:41:33.635840 15757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:33.635415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.1:40213)
0927 02:41:33.635688 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:33.635747 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":76,"threads_started":1}
I20260927 02:41:33.635872 15758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:33.635419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.3:37761)
0927 02:41:33.635739 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:33.635789 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":78,"threads_started":1}
I20260927 02:41:33.829061 15759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:33.828654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.2:55059)
0927 02:41:33.828911 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:33.828968 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":83,"threads_started":1}
I20260927 02:41:34.636920 15760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:34.636539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.1:40431)
0927 02:41:34.636778 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:34.636829 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":73,"threads_started":1}
I20260927 02:41:34.636971 15761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:34.636598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.3:40553)
0927 02:41:34.636829 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:34.636881 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":70,"threads_started":1}
I20260927 02:41:34.830204 15762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:34.829805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.2:53077)
0927 02:41:34.830058 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:34.830114 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":67,"threads_started":1}
I20260927 02:41:35.638278 15763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:35.637797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.1:40563)
0927 02:41:35.638090 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:35.638148 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":70,"threads_started":1}
I20260927 02:41:35.638283 15764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:35.637807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.3:47665)
0927 02:41:35.638110 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:35.638163 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":93,"threads_started":1}
I20260927 02:41:35.831287 15765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:35.830892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.2:35601)
0927 02:41:35.831140 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:35.831198 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":77,"threads_started":1}
I20260927 02:41:36.640092 15766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:36.639081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.3:50995)
0927 02:41:36.639886 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:36.639962 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":709,"thread_start_us":87,"threads_started":1}
I20260927 02:41:36.643041 15767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:36.641595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.1:54289)
0927 02:41:36.642872 (+ 1277us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:36.642938 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1190,"thread_start_us":75,"threads_started":1}
I20260927 02:41:36.832476 15768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:36.832079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45947 (local address 127.0.98.2:51307)
0927 02:41:36.832323 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:36.832386 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":71,"threads_started":1}
I20260927 02:41:37.206568 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 15269
I20260927 02:41:37.211733 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 15436
I20260927 02:41:37.216148 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 15571
2026-09-27T02:41:37Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19114 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:41:37 dist-test-slave-zv8z krb5kdc[15772](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:41:37 dist-test-slave-zv8z krb5kdc[15772](info): set up 2 sockets
Sep 27 02:41:37 dist-test-slave-zv8z krb5kdc[15772](info): commencing operation
krb5kdc: starting...
W20260927 02:41:38.178205 392 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/test-user.keytab.
Sep 27 02:41:38 dist-test-slave-zv8z krb5kdc[15772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:41:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:41:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:41:38.292760 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42269
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:41:38.397861 15785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:41:38.398141 15785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:41:38.398198 15785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:41:38.402134 15785 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 02:41:38.402191 15785 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 02:41:38.402210 15785 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 02:41:38.402230 15785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:41:38.402271 15785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:41:38.402300 15785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:41:38.402329 15785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:41:38.402351 15785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:41:38.407570 15785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:41:38.408591 15785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:41:38.409615 15785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:41:38.418233 15790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:41:38.418359 15785 server_base.cc:1034] running on GCE node
W20260927 02:41:38.418318 15791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:41:38.418318 15793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:41:38.418962 15785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:41:38.420017 15785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:41:38.421183 15785 hybrid_clock.cc:648] HybridClock initialized: now 1790476898421166 us; error 32 us; skew 500 ppm
Sep 27 02:41:38 dist-test-slave-zv8z krb5kdc[15772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:41:38.423799 15785 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:41:38.424626 15785 webserver.cc:460] Webserver started at http://127.0.98.62:41303/ using document root <none> and password file <none>
I20260927 02:41:38.425146 15785 fs_manager.cc:359] Metadata directory not provided
I20260927 02:41:38.425213 15785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:41:38.425408 15785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:41:38.427372 15785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/master-0/data/instance:
uuid: "57522849be664587a9597a5f173c4fd4"
format_stamp: "Formatted at 2026-09-27 02:41:38 on dist-test-slave-zv8z"
server_key: "abbb70c17717280875fec0f01accbeba"
server_key_iv: "317f0961e59f624e9656eae615c27ab3"
server_key_version: "encryptionkey@0"
I20260927 02:41:38.427922 15785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/master-0/wal/instance:
uuid: "57522849be664587a9597a5f173c4fd4"
format_stamp: "Formatted at 2026-09-27 02:41:38 on dist-test-slave-zv8z"
server_key: "abbb70c17717280875fec0f01accbeba"
server_key_iv: "317f0961e59f624e9656eae615c27ab3"
server_key_version: "encryptionkey@0"
I20260927 02:41:38.431447 15785 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:41:38.433898 15800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:38.435086 15785 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:41:38.435209 15785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "57522849be664587a9597a5f173c4fd4"
format_stamp: "Formatted at 2026-09-27 02:41:38 on dist-test-slave-zv8z"
server_key: "abbb70c17717280875fec0f01accbeba"
server_key_iv: "317f0961e59f624e9656eae615c27ab3"
server_key_version: "encryptionkey@0"
I20260927 02:41:38.435307 15785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:41:38.456090 15785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:41:38.459218 15785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:41:38.459394 15785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:41:38.466223 15785 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42269
I20260927 02:41:38.466532 15852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42269 every 8 connection(s)
I20260927 02:41:38.467267 15785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:41:38.468261 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 15785
I20260927 02:41:38.468390 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/master-0/wal/instance
I20260927 02:41:38.468693 392 external_mini_cluster.cc:1428] Setting key 81915aeb5d3d02225fd4eada30e69490
I20260927 02:41:38.470480 15853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:41:38.476491 15853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4: Bootstrap starting.
Sep 27 02:41:38 dist-test-slave-zv8z krb5kdc[15772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:41:38.478717 15853 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4: Neither blocks nor log segments found. Creating new log.
I20260927 02:41:38.479534 15853 log.cc:824] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4: Log is configured to *not* fsync() on all Append() calls
I20260927 02:41:38.481660 15853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4: No bootstrap required, opened a new log
I20260927 02:41:38.482616 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:38.470007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41802 (local address 127.0.98.62:42269)
0927 02:41:38.470432 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:38.470448 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:38.470484 (+ 36us) server_negotiation.cc:407] Connection header received
0927 02:41:38.471145 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:38.471163 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:38.471483 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:38.471763 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:41:38.472593 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:38.473594 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:38.474729 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:38.475189 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:38.475309 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:38.477852 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:41:38.477858 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:41:38.477870 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:41:38.477898 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:41:38.479720 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:38.480151 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:38.480156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:38.480160 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:38.480284 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:38.480529 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:38.480532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:38.480534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:38.480996 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:41:38.481165 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:38.481404 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 02:41:38.481676 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":137,"threads_started":1}
I20260927 02:41:38.485452 15853 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57522849be664587a9597a5f173c4fd4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42269 } }
I20260927 02:41:38.485651 15853 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:41:38.485689 15853 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57522849be664587a9597a5f173c4fd4, State: Initialized, Role: FOLLOWER
I20260927 02:41:38.486064 15853 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57522849be664587a9597a5f173c4fd4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42269 } }
I20260927 02:41:38.486217 15853 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:41:38.486332 15853 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:41:38.486433 15853 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:41:38.487247 15853 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57522849be664587a9597a5f173c4fd4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42269 } }
I20260927 02:41:38.487545 15853 leader_election.cc:302] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57522849be664587a9597a5f173c4fd4; no voters:
I20260927 02:41:38.487812 15853 leader_election.cc:288] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:41:38.487918 15858 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:41:38.488123 15858 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 57522849be664587a9597a5f173c4fd4, State: Running, Role: LEADER
I20260927 02:41:38.488453 15858 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57522849be664587a9597a5f173c4fd4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42269 } }
I20260927 02:41:38.488847 15853 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:41:38.489882 15860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57522849be664587a9597a5f173c4fd4. Latest consensus state: current_term: 1 leader_uuid: "57522849be664587a9597a5f173c4fd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57522849be664587a9597a5f173c4fd4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42269 } } }
I20260927 02:41:38.490006 15860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4 [sys.catalog]: This master's current role is: LEADER
I20260927 02:41:38.490271 15859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57522849be664587a9597a5f173c4fd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57522849be664587a9597a5f173c4fd4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42269 } } }
I20260927 02:41:38.490398 15859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4 [sys.catalog]: This master's current role is: LEADER
I20260927 02:41:38.490684 15867 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:41:38.493199 15867 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:41:38.497879 15867 catalog_manager.cc:1347] Generated new cluster ID: 967a46d087bf4409992cdc59e4368267
I20260927 02:41:38.497963 15867 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:41:38.504297 15867 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:41:38.505174 15867 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:41:38.523187 15867 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4: Generated new TSK 0
I20260927 02:41:38.523885 15867 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:41:38.560480 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42269
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:41:38.669728 15879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:41:38.669961 15879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:41:38.670006 15879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:41:38.673990 15879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:41:38.674048 15879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:41:38.674108 15879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:41:38.679278 15879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:41:38.680274 15879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:41:38.681245 15879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:41:38.690160 15887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:41:38.690169 15884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:41:38.690274 15885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:41:38.690362 15879 server_base.cc:1034] running on GCE node
I20260927 02:41:38.690874 15879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:41:38.691483 15879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:41:38.692628 15879 hybrid_clock.cc:648] HybridClock initialized: now 1790476898692617 us; error 61 us; skew 500 ppm
Sep 27 02:41:38 dist-test-slave-zv8z krb5kdc[15772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:41:38.695185 15879 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:41:38.695990 15879 webserver.cc:460] Webserver started at http://127.0.98.1:33605/ using document root <none> and password file <none>
I20260927 02:41:38.696504 15879 fs_manager.cc:359] Metadata directory not provided
I20260927 02:41:38.696566 15879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:41:38.696784 15879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:41:38.698786 15879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-0/data/instance:
uuid: "fca1d04e37bd41b4a5792c073f2808ed"
format_stamp: "Formatted at 2026-09-27 02:41:38 on dist-test-slave-zv8z"
server_key: "e94d55e37f959a06487a904c344a2a01"
server_key_iv: "5667ca9bccac7daea5a2c57cfd61c9eb"
server_key_version: "encryptionkey@0"
I20260927 02:41:38.699272 15879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fca1d04e37bd41b4a5792c073f2808ed"
format_stamp: "Formatted at 2026-09-27 02:41:38 on dist-test-slave-zv8z"
server_key: "e94d55e37f959a06487a904c344a2a01"
server_key_iv: "5667ca9bccac7daea5a2c57cfd61c9eb"
server_key_version: "encryptionkey@0"
I20260927 02:41:38.702653 15879 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:41:38.704975 15894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:38.706142 15879 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:41:38.706243 15879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-0/wal
uuid: "fca1d04e37bd41b4a5792c073f2808ed"
format_stamp: "Formatted at 2026-09-27 02:41:38 on dist-test-slave-zv8z"
server_key: "e94d55e37f959a06487a904c344a2a01"
server_key_iv: "5667ca9bccac7daea5a2c57cfd61c9eb"
server_key_version: "encryptionkey@0"
I20260927 02:41:38.706364 15879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:41:38.724298 15879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:41:38.727236 15879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:41:38.727380 15879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:41:38.727885 15879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:41:38.728626 15879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:41:38.728670 15879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:38.728722 15879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:41:38.728742 15879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:38.736807 15879 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39877
I20260927 02:41:38.736833 16007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39877 every 8 connection(s)
I20260927 02:41:38.737684 15879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:41:38.745847 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 15879
I20260927 02:41:38.745994 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:41:38.746320 392 external_mini_cluster.cc:1428] Setting key c3677fc955bfb02c6250ba661e60002b
Sep 27 02:41:38 dist-test-slave-zv8z krb5kdc[15772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:41:38.750389 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:38.739392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52099 (local address 127.0.98.62:42269)
0927 02:41:38.739505 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:38.739509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:38.740585 (+ 1076us) server_negotiation.cc:407] Connection header received
0927 02:41:38.741434 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:38.741438 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:38.741494 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:38.741564 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:41:38.742733 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:38.743256 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:38.744338 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:38.744569 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:38.744655 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:38.747081 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:41:38.747084 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:41:38.747086 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:41:38.747110 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:41:38.748538 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:38.748970 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:38.748973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:38.748975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:38.749055 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:38.749360 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:38.749363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:38.749365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:38.749538 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:41:38.749623 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:38.750086 (+ 463us) server_negotiation.cc:299] Negotiation successful
0927 02:41:38.750208 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 02:41:38.751243 16010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:38.739816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.1:52099)
0927 02:41:38.740446 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:38.740487 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:41:38.741245 (+ 758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:41:38.741749 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:41:38.741756 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:41:38.742087 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:41:38.742595 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:38.742609 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:38.743368 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:38.743371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:38.744228 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:38.744237 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:38.744700 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:38.744704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:38.744857 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:38.745344 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:41:38.745359 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:41:38.746944 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:41:38.748642 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:38.748646 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:38.748657 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:38.748888 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:38.749127 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:38.749130 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:38.749132 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:38.749271 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:38.749634 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:41:38.749641 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:41:38.749873 (+ 232us) client_negotiation.cc:769] Sending connection context
0927 02:41:38.750032 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 02:41:38.750383 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":108,"threads_started":1}
I20260927 02:41:38.752331 16008 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42269
I20260927 02:41:38.752588 16008 heartbeater.cc:464] Registering TS with master...
I20260927 02:41:38.753165 16008 heartbeater.cc:511] Master 127.0.98.62:42269 requested a full tablet report, sending...
I20260927 02:41:38.754666 15817 ts_manager.cc:194] Registered new tserver with Master: fca1d04e37bd41b4a5792c073f2808ed (127.0.98.1:39877)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:41:38.755762 15817 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52099
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:41:38.771275 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42269
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:41:38.876047 16013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:41:38.876296 16013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:41:38.876336 16013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:41:38.880265 16013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:41:38.880317 16013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:41:38.880373 16013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:41:38.885390 16013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:41:38.886379 16013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:41:38.887367 16013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:41:38.896533 16021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:41:38.896553 16019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:41:38.896533 16018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:41:38.896775 16013 server_base.cc:1034] running on GCE node
I20260927 02:41:38.897163 16013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:41:38.897761 16013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:41:38.898910 16013 hybrid_clock.cc:648] HybridClock initialized: now 1790476898898891 us; error 31 us; skew 500 ppm
Sep 27 02:41:38 dist-test-slave-zv8z krb5kdc[15772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:41:38.901470 16013 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:41:38.902315 16013 webserver.cc:460] Webserver started at http://127.0.98.2:34579/ using document root <none> and password file <none>
I20260927 02:41:38.902825 16013 fs_manager.cc:359] Metadata directory not provided
I20260927 02:41:38.902887 16013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:41:38.903065 16013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:41:38.905076 16013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-1/data/instance:
uuid: "e9a7c9d995ee4f51b1f1c8fe9292ba57"
format_stamp: "Formatted at 2026-09-27 02:41:38 on dist-test-slave-zv8z"
server_key: "5353c03c6d9552a836be848f6241225f"
server_key_iv: "a25653b8b69b5d81c5f5b950f8420ded"
server_key_version: "encryptionkey@0"
I20260927 02:41:38.905597 16013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e9a7c9d995ee4f51b1f1c8fe9292ba57"
format_stamp: "Formatted at 2026-09-27 02:41:38 on dist-test-slave-zv8z"
server_key: "5353c03c6d9552a836be848f6241225f"
server_key_iv: "a25653b8b69b5d81c5f5b950f8420ded"
server_key_version: "encryptionkey@0"
I20260927 02:41:38.909035 16013 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260927 02:41:38.911382 16028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:38.912510 16013 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:41:38.912611 16013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-1/wal
uuid: "e9a7c9d995ee4f51b1f1c8fe9292ba57"
format_stamp: "Formatted at 2026-09-27 02:41:38 on dist-test-slave-zv8z"
server_key: "5353c03c6d9552a836be848f6241225f"
server_key_iv: "a25653b8b69b5d81c5f5b950f8420ded"
server_key_version: "encryptionkey@0"
I20260927 02:41:38.912705 16013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:41:38.928905 16013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:41:38.931588 16013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:41:38.931756 16013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:41:38.932305 16013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:41:38.933058 16013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:41:38.933105 16013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:38.933166 16013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:41:38.933192 16013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:38.941529 16013 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35573
I20260927 02:41:38.941556 16141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35573 every 8 connection(s)
I20260927 02:41:38.942430 16013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:41:38.946475 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 16013
I20260927 02:41:38.946581 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:41:38.946846 392 external_mini_cluster.cc:1428] Setting key 7979ea1647bf78821c94aea5486b0875
Sep 27 02:41:38 dist-test-slave-zv8z krb5kdc[15772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:41:38.955026 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:38.944029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34385 (local address 127.0.98.62:42269)
0927 02:41:38.944147 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:38.944150 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:38.945144 (+ 994us) server_negotiation.cc:407] Connection header received
0927 02:41:38.945942 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:38.945944 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:38.945988 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:38.946056 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:41:38.947216 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:38.947813 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:38.948989 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:38.949289 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:38.949382 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:38.951879 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:41:38.951882 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:41:38.951884 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:41:38.951907 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:41:38.953164 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:38.953612 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:38.953616 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:38.953618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:38.953699 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:38.953996 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:38.954000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:38.954002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:38.954180 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:41:38.954289 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:38.954754 (+ 465us) server_negotiation.cc:299] Negotiation successful
0927 02:41:38.954871 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 02:41:38.955797 16144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:38.944409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.2:34385)
0927 02:41:38.945008 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:38.945045 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 02:41:38.945776 (+ 731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:41:38.946209 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:41:38.946215 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:41:38.946592 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:41:38.947081 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:38.947096 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:38.947962 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:38.947966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:38.948873 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:38.948882 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:38.949427 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:38.949431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:38.949620 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:38.950178 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:41:38.950197 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:41:38.951731 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:41:38.953266 (+ 1535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:38.953272 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:38.953287 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:38.953523 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:38.953782 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:38.953785 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:38.953787 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:38.953914 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:38.954295 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:41:38.954302 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:41:38.954531 (+ 229us) client_negotiation.cc:769] Sending connection context
0927 02:41:38.954697 (+ 166us) client_negotiation.cc:240] Negotiation successful
0927 02:41:38.955001 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":104,"threads_started":1}
I20260927 02:41:38.956890 16142 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42269
I20260927 02:41:38.957101 16142 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 02:41:38.957568 16142 heartbeater.cc:511] Master 127.0.98.62:42269 requested a full tablet report, sending...
I20260927 02:41:38.958591 15817 ts_manager.cc:194] Registered new tserver with Master: e9a7c9d995ee4f51b1f1c8fe9292ba57 (127.0.98.2:35573)
I20260927 02:41:38.959167 15817 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34385
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:41:38.971144 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42269
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:41:39.074882 16147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:41:39.075119 16147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:41:39.075157 16147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:41:39.079100 16147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:41:39.079156 16147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:41:39.079210 16147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:41:39.084235 16147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:41:39.085176 16147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:41:39.086143 16147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:41:39.095194 16152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:41:39.095189 16153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:41:39.095194 16155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:41:39.095280 16147 server_base.cc:1034] running on GCE node
I20260927 02:41:39.095924 16147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:41:39.096501 16147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:41:39.097664 16147 hybrid_clock.cc:648] HybridClock initialized: now 1790476899097645 us; error 29 us; skew 500 ppm
Sep 27 02:41:39 dist-test-slave-zv8z krb5kdc[15772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476899, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:41:39.100212 16147 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:41:39.101006 16147 webserver.cc:460] Webserver started at http://127.0.98.3:36161/ using document root <none> and password file <none>
I20260927 02:41:39.101516 16147 fs_manager.cc:359] Metadata directory not provided
I20260927 02:41:39.101577 16147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:41:39.101749 16147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:41:39.103673 16147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-2/data/instance:
uuid: "46370315edec4941bcd60a3c10b2b4a7"
format_stamp: "Formatted at 2026-09-27 02:41:39 on dist-test-slave-zv8z"
server_key: "dafdbd5c0ed0f5a156e001b95997a30c"
server_key_iv: "1a1c2589262cd937ca81ebd0b48ac77f"
server_key_version: "encryptionkey@0"
I20260927 02:41:39.104187 16147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-2/wal/instance:
uuid: "46370315edec4941bcd60a3c10b2b4a7"
format_stamp: "Formatted at 2026-09-27 02:41:39 on dist-test-slave-zv8z"
server_key: "dafdbd5c0ed0f5a156e001b95997a30c"
server_key_iv: "1a1c2589262cd937ca81ebd0b48ac77f"
server_key_version: "encryptionkey@0"
I20260927 02:41:39.107699 16147 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:41:39.110193 16162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:39.111477 16147 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:41:39.111591 16147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-2/wal
uuid: "46370315edec4941bcd60a3c10b2b4a7"
format_stamp: "Formatted at 2026-09-27 02:41:39 on dist-test-slave-zv8z"
server_key: "dafdbd5c0ed0f5a156e001b95997a30c"
server_key_iv: "1a1c2589262cd937ca81ebd0b48ac77f"
server_key_version: "encryptionkey@0"
I20260927 02:41:39.111683 16147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:41:39.129772 16147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:41:39.132838 16147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:41:39.133015 16147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:41:39.133589 16147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:41:39.134406 16147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:41:39.134460 16147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:39.134521 16147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:41:39.134544 16147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:39.143043 16147 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37805
I20260927 02:41:39.143083 16275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37805 every 8 connection(s)
I20260927 02:41:39.143905 16147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:41:39.146587 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 16147
I20260927 02:41:39.146719 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:41:39.147037 392 external_mini_cluster.cc:1428] Setting key f0d7977624fadf8b7cca2b9373bd8926
Sep 27 02:41:39 dist-test-slave-zv8z krb5kdc[15772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476899, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:41:39.157145 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:39.145670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46709 (local address 127.0.98.62:42269)
0927 02:41:39.145787 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:39.145791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:39.146912 (+ 1121us) server_negotiation.cc:407] Connection header received
0927 02:41:39.147907 (+ 995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:39.147911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:39.147991 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:39.148065 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:41:39.149371 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:39.149969 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:39.151199 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:39.151424 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:39.151513 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:39.153874 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:41:39.153877 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:41:39.153879 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:41:39.153904 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:41:39.155293 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:39.155731 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:39.155733 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:39.155734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:39.155805 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:39.156093 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:39.156096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:39.156098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:39.156276 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:41:39.156355 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:39.156874 (+ 519us) server_negotiation.cc:299] Negotiation successful
0927 02:41:39.156975 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 02:41:39.157846 16278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:39.145986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.3:46709)
0927 02:41:39.146755 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:39.146807 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 02:41:39.147715 (+ 908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:41:39.148255 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:41:39.148262 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:41:39.148668 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:41:39.149230 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:39.149250 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:39.150122 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:39.150125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:39.151073 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:39.151082 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:39.151555 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:39.151558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:39.151701 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:39.152234 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:41:39.152251 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:41:39.153720 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:41:39.155397 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:39.155401 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:39.155413 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:39.155654 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:39.155868 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:39.155870 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:39.155871 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:39.155994 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:39.156396 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:41:39.156401 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:41:39.156624 (+ 223us) client_negotiation.cc:769] Sending connection context
0927 02:41:39.156794 (+ 170us) client_negotiation.cc:240] Negotiation successful
0927 02:41:39.157040 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":108,"threads_started":1}
I20260927 02:41:39.158910 16276 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42269
I20260927 02:41:39.159138 16276 heartbeater.cc:464] Registering TS with master...
I20260927 02:41:39.159664 16276 heartbeater.cc:511] Master 127.0.98.62:42269 requested a full tablet report, sending...
I20260927 02:41:39.160705 15816 ts_manager.cc:194] Registered new tserver with Master: 46370315edec4941bcd60a3c10b2b4a7 (127.0.98.3:37805)
I20260927 02:41:39.161273 15816 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46709
I20260927 02:41:39.172145 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:41:39.180423 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:39.173841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41810 (local address 127.0.98.62:42269)
0927 02:41:39.173959 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:39.173964 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:39.174140 (+ 176us) server_negotiation.cc:407] Connection header received
0927 02:41:39.174271 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:39.174274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:39.174317 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:39.174378 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:41:39.175030 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:39.175534 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:39.176316 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:39.176549 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:39.176633 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:39.177876 (+ 1243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:41:39.177880 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:41:39.177882 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:41:39.177906 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:41:39.178945 (+ 1039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:39.179299 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:39.179302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:39.179304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:39.179373 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:39.179599 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:39.179602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:39.179604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:39.179803 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:41:39.179916 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:39.180120 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 02:41:39.180241 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 02:41:39.183055 15816 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41810:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:41:39.185937 15816 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:41:39.200248 16289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:39.194397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37805 (local address 127.0.0.1:56772)
0927 02:41:39.194846 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:39.194859 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:41:39.194941 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:41:39.196447 (+ 1506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:41:39.196450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:41:39.196467 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:41:39.196566 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:39.196573 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:39.197844 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:39.197847 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:39.198983 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:39.198990 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:39.199743 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:39.199746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:39.199844 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:39.199880 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 02:41:39.199981 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 02:41:39.200097 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"mutex_wait_us":8,"thread_start_us":85,"threads_started":1}
I20260927 02:41:39.200258 16291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:39.194831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56772 (local address 127.0.98.3:37805)
0927 02:41:39.196129 (+ 1298us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:39.196137 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:39.196152 (+ 15us) server_negotiation.cc:407] Connection header received
0927 02:41:39.196207 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:39.196210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:39.196327 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:39.196447 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:41:39.196686 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:39.197717 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:39.199107 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:39.199628 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:39.199726 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:39.199871 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:39.200025 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 02:41:39.200118 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1216,"thread_start_us":104,"threads_started":1}
I20260927 02:41:39.200562 16290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:39.194852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39877 (local address 127.0.0.1:55112)
0927 02:41:39.195103 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:39.195112 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:41:39.195174 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:41:39.196050 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:41:39.196054 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:41:39.196072 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:41:39.196164 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:39.196169 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:39.197467 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:39.197470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:39.198939 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:39.198949 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:39.200057 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:39.200061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:39.200233 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:39.200280 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 02:41:39.200346 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 02:41:39.200432 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":83,"thread_start_us":79,"threads_started":1}
I20260927 02:41:39.200675 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:39.194309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38536 (local address 127.0.98.2:35573)
0927 02:41:39.195403 (+ 1094us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:39.195413 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:39.195432 (+ 19us) server_negotiation.cc:407] Connection header received
0927 02:41:39.195503 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:39.195508 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:39.195726 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:39.195881 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:41:39.196799 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:39.197586 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:39.198958 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:39.199752 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:39.199855 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:39.199986 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:39.200431 (+ 445us) server_negotiation.cc:299] Negotiation successful
0927 02:41:39.200537 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1001,"thread_start_us":137,"threads_started":1}
I20260927 02:41:39.200980 16292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:39.195030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55112 (local address 127.0.98.1:39877)
0927 02:41:39.195473 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:39.195485 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:39.195515 (+ 30us) server_negotiation.cc:407] Connection header received
0927 02:41:39.195577 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:39.195581 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:39.195756 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:39.195899 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:41:39.196272 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:39.197322 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:39.199057 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:39.199776 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:39.199857 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:39.199986 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:39.200790 (+ 804us) server_negotiation.cc:299] Negotiation successful
0927 02:41:39.200876 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":97,"threads_started":1}
I20260927 02:41:39.201263 16287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:39.194071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35573 (local address 127.0.0.1:38536)
0927 02:41:39.194443 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:39.194504 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 02:41:39.194633 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:41:39.196610 (+ 1977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:41:39.196613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:41:39.196629 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:41:39.196701 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:39.196706 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:39.197725 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:39.197728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:39.198850 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:39.198858 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:39.200057 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:39.200060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:39.200230 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:39.200284 (+ 54us) client_negotiation.cc:769] Sending connection context
0927 02:41:39.200346 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 02:41:39.201150 (+ 804us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":781,"thread_start_us":86,"threads_started":1}
I20260927 02:41:39.203549 16076 tablet_service.cc:1467] Processing CreateTablet for tablet dc3d784f32f64ec0b8239cf178d81851 (DEFAULT_TABLE table=test-table [id=0a1399ebccf94f0eb6b71d71d79e1e4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:41:39.203519 16210 tablet_service.cc:1467] Processing CreateTablet for tablet dc3d784f32f64ec0b8239cf178d81851 (DEFAULT_TABLE table=test-table [id=0a1399ebccf94f0eb6b71d71d79e1e4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:41:39.203521 15942 tablet_service.cc:1467] Processing CreateTablet for tablet dc3d784f32f64ec0b8239cf178d81851 (DEFAULT_TABLE table=test-table [id=0a1399ebccf94f0eb6b71d71d79e1e4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:41:39.204411 16076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc3d784f32f64ec0b8239cf178d81851. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:41:39.204485 15942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc3d784f32f64ec0b8239cf178d81851. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:41:39.204502 16210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc3d784f32f64ec0b8239cf178d81851. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:41:39.211017 16293 tablet_bootstrap.cc:492] T dc3d784f32f64ec0b8239cf178d81851 P e9a7c9d995ee4f51b1f1c8fe9292ba57: Bootstrap starting.
I20260927 02:41:39.212039 16294 tablet_bootstrap.cc:492] T dc3d784f32f64ec0b8239cf178d81851 P 46370315edec4941bcd60a3c10b2b4a7: Bootstrap starting.
I20260927 02:41:39.213267 16293 tablet_bootstrap.cc:654] T dc3d784f32f64ec0b8239cf178d81851 P e9a7c9d995ee4f51b1f1c8fe9292ba57: Neither blocks nor log segments found. Creating new log.
I20260927 02:41:39.213654 16294 tablet_bootstrap.cc:654] T dc3d784f32f64ec0b8239cf178d81851 P 46370315edec4941bcd60a3c10b2b4a7: Neither blocks nor log segments found. Creating new log.
I20260927 02:41:39.214040 16293 log.cc:824] T dc3d784f32f64ec0b8239cf178d81851 P e9a7c9d995ee4f51b1f1c8fe9292ba57: Log is configured to *not* fsync() on all Append() calls
I20260927 02:41:39.214599 16295 tablet_bootstrap.cc:492] T dc3d784f32f64ec0b8239cf178d81851 P fca1d04e37bd41b4a5792c073f2808ed: Bootstrap starting.
I20260927 02:41:39.214778 16294 log.cc:824] T dc3d784f32f64ec0b8239cf178d81851 P 46370315edec4941bcd60a3c10b2b4a7: Log is configured to *not* fsync() on all Append() calls
I20260927 02:41:39.216141 16293 tablet_bootstrap.cc:492] T dc3d784f32f64ec0b8239cf178d81851 P e9a7c9d995ee4f51b1f1c8fe9292ba57: No bootstrap required, opened a new log
I20260927 02:41:39.216327 16293 ts_tablet_manager.cc:1397] T dc3d784f32f64ec0b8239cf178d81851 P e9a7c9d995ee4f51b1f1c8fe9292ba57: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 02:41:39.216706 16295 tablet_bootstrap.cc:654] T dc3d784f32f64ec0b8239cf178d81851 P fca1d04e37bd41b4a5792c073f2808ed: Neither blocks nor log segments found. Creating new log.
I20260927 02:41:39.216926 16294 tablet_bootstrap.cc:492] T dc3d784f32f64ec0b8239cf178d81851 P 46370315edec4941bcd60a3c10b2b4a7: No bootstrap required, opened a new log
I20260927 02:41:39.217115 16294 ts_tablet_manager.cc:1397] T dc3d784f32f64ec0b8239cf178d81851 P 46370315edec4941bcd60a3c10b2b4a7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 02:41:39.217841 16295 log.cc:824] T dc3d784f32f64ec0b8239cf178d81851 P fca1d04e37bd41b4a5792c073f2808ed: Log is configured to *not* fsync() on all Append() calls
I20260927 02:41:39.219743 16295 tablet_bootstrap.cc:492] T dc3d784f32f64ec0b8239cf178d81851 P fca1d04e37bd41b4a5792c073f2808ed: No bootstrap required, opened a new log
I20260927 02:41:39.219918 16295 ts_tablet_manager.cc:1397] T dc3d784f32f64ec0b8239cf178d81851 P fca1d04e37bd41b4a5792c073f2808ed: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 02:41:39.220067 16293 raft_consensus.cc:348] T dc3d784f32f64ec0b8239cf178d81851 P e9a7c9d995ee4f51b1f1c8fe9292ba57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a7c9d995ee4f51b1f1c8fe9292ba57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35573 } } peers { permanent_uuid: "fca1d04e37bd41b4a5792c073f2808ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39877 } } peers { permanent_uuid: "46370315edec4941bcd60a3c10b2b4a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37805 } }
I20260927 02:41:39.220049 16294 raft_consensus.cc:348] T dc3d784f32f64ec0b8239cf178d81851 P 46370315edec4941bcd60a3c10b2b4a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a7c9d995ee4f51b1f1c8fe9292ba57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35573 } } peers { permanent_uuid: "fca1d04e37bd41b4a5792c073f2808ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39877 } } peers { permanent_uuid: "46370315edec4941bcd60a3c10b2b4a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37805 } }
I20260927 02:41:39.220311 16293 raft_consensus.cc:374] T dc3d784f32f64ec0b8239cf178d81851 P e9a7c9d995ee4f51b1f1c8fe9292ba57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:41:39.220314 16294 raft_consensus.cc:374] T dc3d784f32f64ec0b8239cf178d81851 P 46370315edec4941bcd60a3c10b2b4a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:41:39.220359 16293 raft_consensus.cc:729] T dc3d784f32f64ec0b8239cf178d81851 P e9a7c9d995ee4f51b1f1c8fe9292ba57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9a7c9d995ee4f51b1f1c8fe9292ba57, State: Initialized, Role: FOLLOWER
I20260927 02:41:39.220372 16294 raft_consensus.cc:729] T dc3d784f32f64ec0b8239cf178d81851 P 46370315edec4941bcd60a3c10b2b4a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46370315edec4941bcd60a3c10b2b4a7, State: Initialized, Role: FOLLOWER
I20260927 02:41:39.220770 16293 consensus_queue.cc:260] T dc3d784f32f64ec0b8239cf178d81851 P e9a7c9d995ee4f51b1f1c8fe9292ba57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a7c9d995ee4f51b1f1c8fe9292ba57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35573 } } peers { permanent_uuid: "fca1d04e37bd41b4a5792c073f2808ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39877 } } peers { permanent_uuid: "46370315edec4941bcd60a3c10b2b4a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37805 } }
I20260927 02:41:39.220890 16294 consensus_queue.cc:260] T dc3d784f32f64ec0b8239cf178d81851 P 46370315edec4941bcd60a3c10b2b4a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a7c9d995ee4f51b1f1c8fe9292ba57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35573 } } peers { permanent_uuid: "fca1d04e37bd41b4a5792c073f2808ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39877 } } peers { permanent_uuid: "46370315edec4941bcd60a3c10b2b4a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37805 } }
I20260927 02:41:39.221541 16142 heartbeater.cc:503] Master 127.0.98.62:42269 was elected leader, sending a full tablet report...
I20260927 02:41:39.221832 16294 ts_tablet_manager.cc:1428] T dc3d784f32f64ec0b8239cf178d81851 P 46370315edec4941bcd60a3c10b2b4a7: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 02:41:39.221833 16293 ts_tablet_manager.cc:1428] T dc3d784f32f64ec0b8239cf178d81851 P e9a7c9d995ee4f51b1f1c8fe9292ba57: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 02:41:39.222139 16276 heartbeater.cc:503] Master 127.0.98.62:42269 was elected leader, sending a full tablet report...
I20260927 02:41:39.223610 16295 raft_consensus.cc:348] T dc3d784f32f64ec0b8239cf178d81851 P fca1d04e37bd41b4a5792c073f2808ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a7c9d995ee4f51b1f1c8fe9292ba57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35573 } } peers { permanent_uuid: "fca1d04e37bd41b4a5792c073f2808ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39877 } } peers { permanent_uuid: "46370315edec4941bcd60a3c10b2b4a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37805 } }
I20260927 02:41:39.223825 16295 raft_consensus.cc:374] T dc3d784f32f64ec0b8239cf178d81851 P fca1d04e37bd41b4a5792c073f2808ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:41:39.223877 16295 raft_consensus.cc:729] T dc3d784f32f64ec0b8239cf178d81851 P fca1d04e37bd41b4a5792c073f2808ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fca1d04e37bd41b4a5792c073f2808ed, State: Initialized, Role: FOLLOWER
I20260927 02:41:39.224296 16295 consensus_queue.cc:260] T dc3d784f32f64ec0b8239cf178d81851 P fca1d04e37bd41b4a5792c073f2808ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a7c9d995ee4f51b1f1c8fe9292ba57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35573 } } peers { permanent_uuid: "fca1d04e37bd41b4a5792c073f2808ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39877 } } peers { permanent_uuid: "46370315edec4941bcd60a3c10b2b4a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37805 } }
I20260927 02:41:39.225111 16295 ts_tablet_manager.cc:1428] T dc3d784f32f64ec0b8239cf178d81851 P fca1d04e37bd41b4a5792c073f2808ed: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 02:41:39.225445 16008 heartbeater.cc:503] Master 127.0.98.62:42269 was elected leader, sending a full tablet report...
W20260927 02:41:39.238821 16009 tablet.cc:2369] T dc3d784f32f64ec0b8239cf178d81851 P fca1d04e37bd41b4a5792c073f2808ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:41:39.280447 16301 raft_consensus.cc:482] T dc3d784f32f64ec0b8239cf178d81851 P fca1d04e37bd41b4a5792c073f2808ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:41:39.280668 16301 raft_consensus.cc:504] T dc3d784f32f64ec0b8239cf178d81851 P fca1d04e37bd41b4a5792c073f2808ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a7c9d995ee4f51b1f1c8fe9292ba57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35573 } } peers { permanent_uuid: "fca1d04e37bd41b4a5792c073f2808ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39877 } } peers { permanent_uuid: "46370315edec4941bcd60a3c10b2b4a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37805 } }
I20260927 02:41:39.281774 16301 leader_election.cc:288] T dc3d784f32f64ec0b8239cf178d81851 P fca1d04e37bd41b4a5792c073f2808ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9a7c9d995ee4f51b1f1c8fe9292ba57 (127.0.98.2:35573), 46370315edec4941bcd60a3c10b2b4a7 (127.0.98.3:37805)
I20260927 02:41:39.285400 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:39.282142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41673 (local address 127.0.98.2:35573)
0927 02:41:39.282294 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:39.282299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:39.282521 (+ 222us) server_negotiation.cc:407] Connection header received
0927 02:41:39.282648 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:39.282651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:39.282692 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:39.282753 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:41:39.282991 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:39.283488 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:39.284528 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:39.284914 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:39.285003 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:39.285064 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:39.285223 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 02:41:39.285286 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 02:41:39.285419 16303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:39.282006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35573 (local address 127.0.98.1:41673)
0927 02:41:39.282422 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:39.282433 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:41:39.282536 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:41:39.282774 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:41:39.282777 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:41:39.282790 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:41:39.282895 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:39.282899 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:39.283588 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:39.283591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:39.284413 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:39.284421 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:39.285022 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:39.285025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:39.285114 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:39.285134 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 02:41:39.285196 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 02:41:39.285262 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":6,"thread_start_us":136,"threads_started":1}
I20260927 02:41:39.285619 16291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:39.282116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51325 (local address 127.0.98.3:37805)
0927 02:41:39.282320 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:39.282325 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:39.282575 (+ 250us) server_negotiation.cc:407] Connection header received
0927 02:41:39.282715 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:39.282718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:39.282760 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:39.282819 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:41:39.283056 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:39.283663 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:39.284731 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:39.285130 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:39.285219 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:39.285276 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:39.285455 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 02:41:39.285520 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260927 02:41:39.285637 16302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:39.282006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37805 (local address 127.0.98.1:51325)
0927 02:41:39.282502 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:39.282515 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:41:39.282618 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:41:39.282839 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:41:39.282842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:41:39.282856 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:41:39.282961 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:39.282965 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:39.283789 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:39.283793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:39.284613 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:39.284622 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:39.285241 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:39.285244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:39.285343 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:39.285360 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 02:41:39.285421 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 02:41:39.285484 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"mutex_wait_us":70,"thread_start_us":120,"threads_started":1}
I20260927 02:41:39.286100 16096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc3d784f32f64ec0b8239cf178d81851" candidate_uuid: "fca1d04e37bd41b4a5792c073f2808ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9a7c9d995ee4f51b1f1c8fe9292ba57" is_pre_election: true
I20260927 02:41:39.286412 16096 raft_consensus.cc:2454] T dc3d784f32f64ec0b8239cf178d81851 P e9a7c9d995ee4f51b1f1c8fe9292ba57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fca1d04e37bd41b4a5792c073f2808ed in term 0.
I20260927 02:41:39.286342 16230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc3d784f32f64ec0b8239cf178d81851" candidate_uuid: "fca1d04e37bd41b4a5792c073f2808ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46370315edec4941bcd60a3c10b2b4a7" is_pre_election: true
I20260927 02:41:39.286631 16230 raft_consensus.cc:2454] T dc3d784f32f64ec0b8239cf178d81851 P 46370315edec4941bcd60a3c10b2b4a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fca1d04e37bd41b4a5792c073f2808ed in term 0.
I20260927 02:41:39.286844 15897 leader_election.cc:302] T dc3d784f32f64ec0b8239cf178d81851 P fca1d04e37bd41b4a5792c073f2808ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e9a7c9d995ee4f51b1f1c8fe9292ba57, fca1d04e37bd41b4a5792c073f2808ed; no voters:
I20260927 02:41:39.287128 16301 raft_consensus.cc:2781] T dc3d784f32f64ec0b8239cf178d81851 P fca1d04e37bd41b4a5792c073f2808ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:41:39.287209 16301 raft_consensus.cc:482] T dc3d784f32f64ec0b8239cf178d81851 P fca1d04e37bd41b4a5792c073f2808ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:41:39.287281 16301 raft_consensus.cc:3037] T dc3d784f32f64ec0b8239cf178d81851 P fca1d04e37bd41b4a5792c073f2808ed [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:41:39.288219 16301 raft_consensus.cc:504] T dc3d784f32f64ec0b8239cf178d81851 P fca1d04e37bd41b4a5792c073f2808ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a7c9d995ee4f51b1f1c8fe9292ba57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35573 } } peers { permanent_uuid: "fca1d04e37bd41b4a5792c073f2808ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39877 } } peers { permanent_uuid: "46370315edec4941bcd60a3c10b2b4a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37805 } }
I20260927 02:41:39.288532 16301 leader_election.cc:288] T dc3d784f32f64ec0b8239cf178d81851 P fca1d04e37bd41b4a5792c073f2808ed [CANDIDATE]: Term 1 election: Requested vote from peers e9a7c9d995ee4f51b1f1c8fe9292ba57 (127.0.98.2:35573), 46370315edec4941bcd60a3c10b2b4a7 (127.0.98.3:37805)
I20260927 02:41:39.288826 16096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc3d784f32f64ec0b8239cf178d81851" candidate_uuid: "fca1d04e37bd41b4a5792c073f2808ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9a7c9d995ee4f51b1f1c8fe9292ba57"
I20260927 02:41:39.288893 16230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc3d784f32f64ec0b8239cf178d81851" candidate_uuid: "fca1d04e37bd41b4a5792c073f2808ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46370315edec4941bcd60a3c10b2b4a7"
I20260927 02:41:39.288959 16096 raft_consensus.cc:3037] T dc3d784f32f64ec0b8239cf178d81851 P e9a7c9d995ee4f51b1f1c8fe9292ba57 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:41:39.289019 16230 raft_consensus.cc:3037] T dc3d784f32f64ec0b8239cf178d81851 P 46370315edec4941bcd60a3c10b2b4a7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:41:39.290237 16096 raft_consensus.cc:2454] T dc3d784f32f64ec0b8239cf178d81851 P e9a7c9d995ee4f51b1f1c8fe9292ba57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fca1d04e37bd41b4a5792c073f2808ed in term 1.
I20260927 02:41:39.290237 16230 raft_consensus.cc:2454] T dc3d784f32f64ec0b8239cf178d81851 P 46370315edec4941bcd60a3c10b2b4a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fca1d04e37bd41b4a5792c073f2808ed in term 1.
I20260927 02:41:39.290534 15897 leader_election.cc:302] T dc3d784f32f64ec0b8239cf178d81851 P fca1d04e37bd41b4a5792c073f2808ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e9a7c9d995ee4f51b1f1c8fe9292ba57, fca1d04e37bd41b4a5792c073f2808ed; no voters:
I20260927 02:41:39.290673 16301 raft_consensus.cc:2781] T dc3d784f32f64ec0b8239cf178d81851 P fca1d04e37bd41b4a5792c073f2808ed [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:41:39.290884 16301 raft_consensus.cc:686] T dc3d784f32f64ec0b8239cf178d81851 P fca1d04e37bd41b4a5792c073f2808ed [term 1 LEADER]: Becoming Leader. State: Replica: fca1d04e37bd41b4a5792c073f2808ed, State: Running, Role: LEADER
I20260927 02:41:39.291208 16301 consensus_queue.cc:237] T dc3d784f32f64ec0b8239cf178d81851 P fca1d04e37bd41b4a5792c073f2808ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a7c9d995ee4f51b1f1c8fe9292ba57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35573 } } peers { permanent_uuid: "fca1d04e37bd41b4a5792c073f2808ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39877 } } peers { permanent_uuid: "46370315edec4941bcd60a3c10b2b4a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37805 } }
I20260927 02:41:39.294023 15816 catalog_manager.cc:5495] T dc3d784f32f64ec0b8239cf178d81851 P fca1d04e37bd41b4a5792c073f2808ed reported cstate change: term changed from 0 to 1, leader changed from <none> to fca1d04e37bd41b4a5792c073f2808ed (127.0.98.1). New cstate: current_term: 1 leader_uuid: "fca1d04e37bd41b4a5792c073f2808ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a7c9d995ee4f51b1f1c8fe9292ba57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fca1d04e37bd41b4a5792c073f2808ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46370315edec4941bcd60a3c10b2b4a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37805 } health_report { overall_health: UNKNOWN } } }
I20260927 02:41:39.325388 16292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:39.322000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55124 (local address 127.0.98.1:39877)
0927 02:41:39.322146 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:39.322151 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:39.322165 (+ 14us) server_negotiation.cc:407] Connection header received
0927 02:41:39.322270 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:39.322272 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:39.322328 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:39.322418 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:41:39.322696 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:39.323200 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:39.324024 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:39.324243 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:39.324330 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:39.324618 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:41:39.325020 (+ 402us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:41:39.325136 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:39.325199 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 02:41:39.325254 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 02:41:39.330811 16096 raft_consensus.cc:1261] T dc3d784f32f64ec0b8239cf178d81851 P e9a7c9d995ee4f51b1f1c8fe9292ba57 [term 1 FOLLOWER]: Refusing update from remote peer fca1d04e37bd41b4a5792c073f2808ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:41:39.330812 16230 raft_consensus.cc:1261] T dc3d784f32f64ec0b8239cf178d81851 P 46370315edec4941bcd60a3c10b2b4a7 [term 1 FOLLOWER]: Refusing update from remote peer fca1d04e37bd41b4a5792c073f2808ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:41:39.331454 16301 consensus_queue.cc:1035] T dc3d784f32f64ec0b8239cf178d81851 P fca1d04e37bd41b4a5792c073f2808ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a7c9d995ee4f51b1f1c8fe9292ba57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:41:39.331640 16304 consensus_queue.cc:1035] T dc3d784f32f64ec0b8239cf178d81851 P fca1d04e37bd41b4a5792c073f2808ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "46370315edec4941bcd60a3c10b2b4a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:41:39.338398 16307 mvcc.cc:205] Tried to move back new op lower bound from 7333793379652755456 to 7333793379499311104. Current Snapshot: MvccSnapshot[applied={T|T < 7333793379652755456}]
I20260927 02:41:39.342340 16309 mvcc.cc:205] Tried to move back new op lower bound from 7333793379652755456 to 7333793379499311104. Current Snapshot: MvccSnapshot[applied={T|T < 7333793379652755456}]
I20260927 02:41:39.346500 15817 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:41:39.346643 15817 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:41:39.348857 15817 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4: Sending DeleteTablet for 3 replicas of tablet dc3d784f32f64ec0b8239cf178d81851
I20260927 02:41:39.349663 16210 tablet_service.cc:1514] Processing DeleteTablet for tablet dc3d784f32f64ec0b8239cf178d81851 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:41:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56772
I20260927 02:41:39.349787 16076 tablet_service.cc:1514] Processing DeleteTablet for tablet dc3d784f32f64ec0b8239cf178d81851 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:41:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38536
I20260927 02:41:39.349790 15942 tablet_service.cc:1514] Processing DeleteTablet for tablet dc3d784f32f64ec0b8239cf178d81851 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:41:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55112
I20260927 02:41:39.350167 16316 tablet_replica.cc:330] T dc3d784f32f64ec0b8239cf178d81851 P 46370315edec4941bcd60a3c10b2b4a7: stopping tablet replica
I20260927 02:41:39.350358 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 15879
I20260927 02:41:39.351550 16317 tablet_replica.cc:330] T dc3d784f32f64ec0b8239cf178d81851 P e9a7c9d995ee4f51b1f1c8fe9292ba57: stopping tablet replica
I20260927 02:41:39.351800 16317 raft_consensus.cc:2229] T dc3d784f32f64ec0b8239cf178d81851 P e9a7c9d995ee4f51b1f1c8fe9292ba57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:41:39.352064 16317 raft_consensus.cc:2258] T dc3d784f32f64ec0b8239cf178d81851 P e9a7c9d995ee4f51b1f1c8fe9292ba57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:41:39.352295 16316 raft_consensus.cc:2229] T dc3d784f32f64ec0b8239cf178d81851 P 46370315edec4941bcd60a3c10b2b4a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:41:39.352574 16316 raft_consensus.cc:2258] T dc3d784f32f64ec0b8239cf178d81851 P 46370315edec4941bcd60a3c10b2b4a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:41:39.353436 16317 ts_tablet_manager.cc:1905] T dc3d784f32f64ec0b8239cf178d81851 P e9a7c9d995ee4f51b1f1c8fe9292ba57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:41:39.353925 16316 ts_tablet_manager.cc:1905] T dc3d784f32f64ec0b8239cf178d81851 P 46370315edec4941bcd60a3c10b2b4a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:41:39.356144 16317 ts_tablet_manager.cc:1918] T dc3d784f32f64ec0b8239cf178d81851 P e9a7c9d995ee4f51b1f1c8fe9292ba57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:41:39.356240 16317 log.cc:1196] T dc3d784f32f64ec0b8239cf178d81851 P e9a7c9d995ee4f51b1f1c8fe9292ba57: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-1/wal/wals/dc3d784f32f64ec0b8239cf178d81851
I20260927 02:41:39.356417 16316 ts_tablet_manager.cc:1918] T dc3d784f32f64ec0b8239cf178d81851 P 46370315edec4941bcd60a3c10b2b4a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:41:39.356498 16316 log.cc:1196] T dc3d784f32f64ec0b8239cf178d81851 P 46370315edec4941bcd60a3c10b2b4a7: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-2/wal/wals/dc3d784f32f64ec0b8239cf178d81851
W20260927 02:41:39.356501 15803 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39877 (error 108)
I20260927 02:41:39.356709 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 16013
I20260927 02:41:39.357204 16316 ts_tablet_manager.cc:1939] T dc3d784f32f64ec0b8239cf178d81851 P 46370315edec4941bcd60a3c10b2b4a7: Deleting consensus metadata
I20260927 02:41:39.357908 16290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:39.357589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39877 (local address 127.0.0.1:55128)
0927 02:41:39.357753 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:39.357824 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260927 02:41:39.358062 15803 catalog_manager.cc:4584] TS fca1d04e37bd41b4a5792c073f2808ed (127.0.98.1:39877): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dc3d784f32f64ec0b8239cf178d81851: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39877: connect: Connection refused (error 111)
I20260927 02:41:39.358443 15801 catalog_manager.cc:4841] TS 46370315edec4941bcd60a3c10b2b4a7 (127.0.98.3:37805): tablet dc3d784f32f64ec0b8239cf178d81851 (table test-table [id=0a1399ebccf94f0eb6b71d71d79e1e4e]) successfully deleted
I20260927 02:41:39.362026 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 16147
I20260927 02:41:39.362471 16290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:39.362184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35573 (local address 127.0.0.1:38540)
0927 02:41:39.362354 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:39.362399 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260927 02:41:39.366487 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 15785
I20260927 02:41:59.371949 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42269
--webserver_interface=127.0.98.62
--webserver_port=41303
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:41:59.481242 16320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:41:59.481494 16320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:41:59.481535 16320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:41:59.485492 16320 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 02:41:59.485540 16320 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 02:41:59.485555 16320 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 02:41:59.485567 16320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:41:59.485584 16320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:41:59.485600 16320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:41:59.485627 16320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:41:59.485644 16320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:41:59.490886 16320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:41:59.491870 16320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:41:59.492823 16320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:41:59.505227 16326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:41:59.505347 16320 server_base.cc:1034] running on GCE node
W20260927 02:41:59.505179 16327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:41:59.505249 16329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:41:59.506024 16320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:41:59.507445 16320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:41:59.508625 16320 hybrid_clock.cc:648] HybridClock initialized: now 1790476919508594 us; error 43 us; skew 500 ppm
Sep 27 02:41:59 dist-test-slave-zv8z krb5kdc[15772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:41:59.516362 16320 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:41:59.517316 16320 webserver.cc:460] Webserver started at http://127.0.98.62:41303/ using document root <none> and password file <none>
I20260927 02:41:59.517899 16320 fs_manager.cc:359] Metadata directory not provided
I20260927 02:41:59.517978 16320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:41:59.521868 16320 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:41:59.524114 16336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:59.525333 16320 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 02:41:59.525457 16320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "57522849be664587a9597a5f173c4fd4"
format_stamp: "Formatted at 2026-09-27 02:41:38 on dist-test-slave-zv8z"
server_key: "abbb70c17717280875fec0f01accbeba"
server_key_iv: "317f0961e59f624e9656eae615c27ab3"
server_key_version: "encryptionkey@0"
I20260927 02:41:59.526057 16320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:41:59.539063 16320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:41:59.541890 16320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:41:59.542086 16320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:41:59.548751 16320 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42269
I20260927 02:41:59.549006 16388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42269 every 8 connection(s)
I20260927 02:41:59.549800 16320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:41:59.554035 16389 sys_catalog.cc:263] Verifying existing consensus state
I20260927 02:41:59.555716 16389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4: Bootstrap starting.
I20260927 02:41:59.558447 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 16320
I20260927 02:41:59.558830 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39877
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42269
--webserver_port=33605
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:41:59.572639 16389 log.cc:824] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4: Log is configured to *not* fsync() on all Append() calls
I20260927 02:41:59.581245 16389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:41:59.581576 16389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4: Bootstrap complete.
I20260927 02:41:59.585348 16389 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57522849be664587a9597a5f173c4fd4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42269 } }
I20260927 02:41:59.585731 16389 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57522849be664587a9597a5f173c4fd4, State: Initialized, Role: FOLLOWER
I20260927 02:41:59.586191 16389 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57522849be664587a9597a5f173c4fd4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42269 } }
I20260927 02:41:59.586335 16389 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:41:59.586412 16389 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:41:59.586500 16389 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4 [term 1 FOLLOWER]: Advancing to term 2
I20260927 02:41:59.588316 16389 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57522849be664587a9597a5f173c4fd4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42269 } }
I20260927 02:41:59.588624 16389 leader_election.cc:302] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57522849be664587a9597a5f173c4fd4; no voters:
I20260927 02:41:59.588899 16389 leader_election.cc:288] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 02:41:59.588996 16393 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 02:41:59.589188 16393 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4 [term 2 LEADER]: Becoming Leader. State: Replica: 57522849be664587a9597a5f173c4fd4, State: Running, Role: LEADER
I20260927 02:41:59.589534 16393 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57522849be664587a9597a5f173c4fd4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42269 } }
I20260927 02:41:59.590108 16389 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:41:59.590972 16395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57522849be664587a9597a5f173c4fd4. Latest consensus state: current_term: 2 leader_uuid: "57522849be664587a9597a5f173c4fd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57522849be664587a9597a5f173c4fd4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42269 } } }
I20260927 02:41:59.591007 16394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "57522849be664587a9597a5f173c4fd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57522849be664587a9597a5f173c4fd4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42269 } } }
I20260927 02:41:59.591096 16395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4 [sys.catalog]: This master's current role is: LEADER
I20260927 02:41:59.591112 16394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4 [sys.catalog]: This master's current role is: LEADER
I20260927 02:41:59.594895 16408 catalog_manager.cc:1259] Loaded cluster ID: 967a46d087bf4409992cdc59e4368267
I20260927 02:41:59.594954 16408 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4: loading cluster ID for follower catalog manager: success
I20260927 02:41:59.597375 16408 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4: acquiring CA information for follower catalog manager: success
I20260927 02:41:59.598896 16408 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 02:41:59.599020 16402 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:41:59.601308 16402 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:41:59.601806 16402 catalog_manager.cc:1259] Loaded cluster ID: 967a46d087bf4409992cdc59e4368267
I20260927 02:41:59.601869 16402 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:41:59.602557 16402 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:41:59.603029 16402 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4: Loaded TSK: 0
I20260927 02:41:59.619050 16402 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4: Generated new TSK 1
I20260927 02:41:59.619293 16402 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 02:41:59.672586 16391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:41:59.672894 16391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:41:59.672959 16391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:41:59.677301 16391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:41:59.677357 16391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:41:59.677439 16391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:41:59.682618 16391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39877
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33605
--tserver_master_addrs=127.0.98.62:42269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:41:59.683591 16391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:41:59.684600 16391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:41:59.694707 16416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:41:59.694736 16417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:41:59.694861 16391 server_base.cc:1034] running on GCE node
W20260927 02:41:59.694934 16419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:41:59.695418 16391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:41:59.696192 16391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:41:59.697378 16391 hybrid_clock.cc:648] HybridClock initialized: now 1790476919697361 us; error 28 us; skew 500 ppm
Sep 27 02:41:59 dist-test-slave-zv8z krb5kdc[15772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:41:59.700637 16391 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:41:59.701713 16391 webserver.cc:460] Webserver started at http://127.0.98.1:33605/ using document root <none> and password file <none>
I20260927 02:41:59.702417 16391 fs_manager.cc:359] Metadata directory not provided
I20260927 02:41:59.702517 16391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:41:59.707360 16391 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:41:59.710021 16426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:59.711568 16391 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 02:41:59.711701 16391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-0/wal
uuid: "fca1d04e37bd41b4a5792c073f2808ed"
format_stamp: "Formatted at 2026-09-27 02:41:38 on dist-test-slave-zv8z"
server_key: "e94d55e37f959a06487a904c344a2a01"
server_key_iv: "5667ca9bccac7daea5a2c57cfd61c9eb"
server_key_version: "encryptionkey@0"
I20260927 02:41:59.712391 16391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:41:59.755201 16391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:41:59.758816 16391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:41:59.758999 16391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:41:59.759680 16391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:41:59.760936 16433 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 02:41:59.765069 16391 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 02:41:59.765137 16391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260927 02:41:59.765188 16391 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 02:41:59.766870 16391 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 02:41:59.766913 16391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260927 02:41:59.767194 16433 tablet_bootstrap.cc:492] T dc3d784f32f64ec0b8239cf178d81851 P fca1d04e37bd41b4a5792c073f2808ed: Bootstrap starting.
I20260927 02:41:59.776286 16391 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39877
I20260927 02:41:59.776362 16540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39877 every 8 connection(s)
I20260927 02:41:59.777213 16391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:41:59.778074 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 16391
I20260927 02:41:59.778630 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35573
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42269
--webserver_port=34579
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:41:59.786536 16433 log.cc:824] T dc3d784f32f64ec0b8239cf178d81851 P fca1d04e37bd41b4a5792c073f2808ed: Log is configured to *not* fsync() on all Append() calls
Sep 27 02:41:59 dist-test-slave-zv8z krb5kdc[15772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:41:59.794929 16433 tablet_bootstrap.cc:492] T dc3d784f32f64ec0b8239cf178d81851 P fca1d04e37bd41b4a5792c073f2808ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:41:59.795341 16433 tablet_bootstrap.cc:492] T dc3d784f32f64ec0b8239cf178d81851 P fca1d04e37bd41b4a5792c073f2808ed: Bootstrap complete.
I20260927 02:41:59.796053 16543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:59.779241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54423 (local address 127.0.98.62:42269)
0927 02:41:59.779640 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:59.779659 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:59.780095 (+ 436us) server_negotiation.cc:407] Connection header received
0927 02:41:59.781245 (+ 1150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:59.781256 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:59.781552 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:59.781857 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:41:59.783253 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:59.784750 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:59.786481 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:59.787129 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:59.787306 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:59.790212 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:41:59.790218 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:41:59.790235 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:41:59.790294 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:41:59.792340 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:59.792943 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:59.792949 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:59.792954 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:59.793104 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:59.793519 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:59.793522 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:59.793524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:59.794046 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:41:59.794185 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:59.794826 (+ 641us) server_negotiation.cc:299] Negotiation successful
0927 02:41:59.795171 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":117,"threads_started":1}
I20260927 02:41:59.796082 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:59.779215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.1:54423)
0927 02:41:59.779925 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:59.779977 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 02:41:59.780845 (+ 868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:41:59.781940 (+ 1095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:41:59.781948 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:41:59.782448 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:41:59.783037 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:59.783056 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:59.784924 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:59.784928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:59.786356 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:59.786368 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:59.787281 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:59.787285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:59.787471 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:59.788129 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:41:59.788149 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:41:59.789978 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:41:59.792469 (+ 2491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:59.792476 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:59.792489 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:59.792823 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:59.793234 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:59.793238 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:59.793239 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:59.793407 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:59.794175 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:41:59.794182 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:41:59.794507 (+ 325us) client_negotiation.cc:769] Sending connection context
0927 02:41:59.794714 (+ 207us) client_negotiation.cc:240] Negotiation successful
0927 02:41:59.795133 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":111,"threads_started":1}
I20260927 02:41:59.796591 16433 ts_tablet_manager.cc:1397] T dc3d784f32f64ec0b8239cf178d81851 P fca1d04e37bd41b4a5792c073f2808ed: Time spent bootstrapping tablet: real 0.030s user 0.018s sys 0.008s
I20260927 02:41:59.799152 16541 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42269
I20260927 02:41:59.799386 16541 heartbeater.cc:464] Registering TS with master...
I20260927 02:41:59.799472 16433 raft_consensus.cc:348] T dc3d784f32f64ec0b8239cf178d81851 P fca1d04e37bd41b4a5792c073f2808ed [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a7c9d995ee4f51b1f1c8fe9292ba57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35573 } } peers { permanent_uuid: "fca1d04e37bd41b4a5792c073f2808ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39877 } } peers { permanent_uuid: "46370315edec4941bcd60a3c10b2b4a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37805 } }
I20260927 02:41:59.799882 16433 raft_consensus.cc:729] T dc3d784f32f64ec0b8239cf178d81851 P fca1d04e37bd41b4a5792c073f2808ed [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fca1d04e37bd41b4a5792c073f2808ed, State: Initialized, Role: FOLLOWER
I20260927 02:41:59.799896 16541 heartbeater.cc:511] Master 127.0.98.62:42269 requested a full tablet report, sending...
I20260927 02:41:59.800348 16433 consensus_queue.cc:260] T dc3d784f32f64ec0b8239cf178d81851 P fca1d04e37bd41b4a5792c073f2808ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9a7c9d995ee4f51b1f1c8fe9292ba57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35573 } } peers { permanent_uuid: "fca1d04e37bd41b4a5792c073f2808ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39877 } } peers { permanent_uuid: "46370315edec4941bcd60a3c10b2b4a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37805 } }
I20260927 02:41:59.801242 16433 ts_tablet_manager.cc:1428] T dc3d784f32f64ec0b8239cf178d81851 P fca1d04e37bd41b4a5792c073f2808ed: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260927 02:41:59.802964 16353 ts_manager.cc:194] Registered new tserver with Master: fca1d04e37bd41b4a5792c073f2808ed (127.0.98.1:39877)
I20260927 02:41:59.806349 16353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54423
I20260927 02:41:59.808837 16541 heartbeater.cc:503] Master 127.0.98.62:42269 was elected leader, sending a full tablet report...
Sep 27 02:41:59 dist-test-slave-zv8z krb5kdc[15772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:41:59.815996 16548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:59.805648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39877 (local address 127.0.0.1:57400)
0927 02:41:59.806404 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:59.806450 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 02:41:59.806635 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:41:59.807324 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:41:59.807329 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:41:59.807637 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:41:59.807736 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:59.807743 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:59.808854 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:59.808858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:59.809654 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:59.809664 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:59.810376 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:59.810395 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:59.810501 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:59.811005 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:41:59.811016 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:41:59.812505 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:41:59.814200 (+ 1695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:59.814209 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:59.814215 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:59.814540 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:59.814860 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:59.814862 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:59.814865 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:59.814959 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:59.815356 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:41:59.815362 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:41:59.815482 (+ 120us) client_negotiation.cc:769] Sending connection context
0927 02:41:59.815622 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 02:41:59.815809 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":111,"threads_started":1}
I20260927 02:41:59.816010 16549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:59.805739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57400 (local address 127.0.98.1:39877)
0927 02:41:59.806118 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:59.806129 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:59.806532 (+ 403us) server_negotiation.cc:407] Connection header received
0927 02:41:59.806796 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:59.806802 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:59.807000 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:59.807131 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:41:59.807854 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:59.808750 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:59.809787 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:59.810287 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:59.810388 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:59.812634 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:41:59.812643 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:41:59.812652 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:41:59.812697 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:41:59.814102 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:59.814674 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:59.814679 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:59.814680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:59.814770 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:59.815039 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:59.815041 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:59.815042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:59.815258 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:41:59.815359 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:59.815735 (+ 376us) server_negotiation.cc:299] Negotiation successful
0927 02:41:59.815879 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":101,"threads_started":1}
I20260927 02:41:59.817097 16475 tablet_service.cc:1514] Processing DeleteTablet for tablet dc3d784f32f64ec0b8239cf178d81851 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:41:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57400
I20260927 02:41:59.817965 16550 tablet_replica.cc:330] stopping tablet replica
I20260927 02:41:59.818359 16550 raft_consensus.cc:2229] T dc3d784f32f64ec0b8239cf178d81851 P fca1d04e37bd41b4a5792c073f2808ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:41:59.818612 16550 raft_consensus.cc:2258] T dc3d784f32f64ec0b8239cf178d81851 P fca1d04e37bd41b4a5792c073f2808ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:41:59.819664 16550 ts_tablet_manager.cc:1905] T dc3d784f32f64ec0b8239cf178d81851 P fca1d04e37bd41b4a5792c073f2808ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:41:59.823942 16550 ts_tablet_manager.cc:1918] T dc3d784f32f64ec0b8239cf178d81851 P fca1d04e37bd41b4a5792c073f2808ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:41:59.824038 16550 log.cc:1196] T dc3d784f32f64ec0b8239cf178d81851 P fca1d04e37bd41b4a5792c073f2808ed: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-0/wal/wals/dc3d784f32f64ec0b8239cf178d81851
I20260927 02:41:59.824314 16550 ts_tablet_manager.cc:1939] T dc3d784f32f64ec0b8239cf178d81851 P fca1d04e37bd41b4a5792c073f2808ed: Deleting consensus metadata
I20260927 02:41:59.826354 16339 catalog_manager.cc:4841] TS fca1d04e37bd41b4a5792c073f2808ed (127.0.98.1:39877): tablet dc3d784f32f64ec0b8239cf178d81851 (table test-table [id=0a1399ebccf94f0eb6b71d71d79e1e4e]) successfully deleted
W20260927 02:41:59.895977 16545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:41:59.896224 16545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:41:59.896265 16545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:41:59.900230 16545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:41:59.900291 16545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:41:59.900347 16545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:41:59.905491 16545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35573
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34579
--tserver_master_addrs=127.0.98.62:42269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:41:59.906527 16545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:41:59.907491 16545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:41:59.916654 16556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:41:59.916744 16555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:41:59.916744 16545 server_base.cc:1034] running on GCE node
W20260927 02:41:59.916885 16558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:41:59.917357 16545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:41:59.917971 16545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:41:59.919137 16545 hybrid_clock.cc:648] HybridClock initialized: now 1790476919919084 us; error 68 us; skew 500 ppm
Sep 27 02:41:59 dist-test-slave-zv8z krb5kdc[15772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:41:59.921729 16545 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:41:59.922551 16545 webserver.cc:460] Webserver started at http://127.0.98.2:34579/ using document root <none> and password file <none>
I20260927 02:41:59.923049 16545 fs_manager.cc:359] Metadata directory not provided
I20260927 02:41:59.923123 16545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:41:59.926753 16545 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:41:59.928783 16565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:59.930001 16545 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:41:59.930110 16545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-1/wal
uuid: "e9a7c9d995ee4f51b1f1c8fe9292ba57"
format_stamp: "Formatted at 2026-09-27 02:41:38 on dist-test-slave-zv8z"
server_key: "5353c03c6d9552a836be848f6241225f"
server_key_iv: "a25653b8b69b5d81c5f5b950f8420ded"
server_key_version: "encryptionkey@0"
I20260927 02:41:59.930661 16545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:41:59.944972 16545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:41:59.947975 16545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:41:59.948118 16545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:41:59.948624 16545 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:41:59.949635 16572 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260927 02:41:59.953397 16572 ts_tablet_manager.cc:1852] T dc3d784f32f64ec0b8239cf178d81851 P e9a7c9d995ee4f51b1f1c8fe9292ba57: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260927 02:41:59.953788 16572 ts_tablet_manager.cc:1905] T dc3d784f32f64ec0b8239cf178d81851 P e9a7c9d995ee4f51b1f1c8fe9292ba57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:41:59.957474 16572 ts_tablet_manager.cc:1918] T dc3d784f32f64ec0b8239cf178d81851 P e9a7c9d995ee4f51b1f1c8fe9292ba57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:41:59.957566 16572 log.cc:1196] T dc3d784f32f64ec0b8239cf178d81851 P e9a7c9d995ee4f51b1f1c8fe9292ba57: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-1/wal/wals/dc3d784f32f64ec0b8239cf178d81851
I20260927 02:41:59.957850 16572 ts_tablet_manager.cc:1939] T dc3d784f32f64ec0b8239cf178d81851 P e9a7c9d995ee4f51b1f1c8fe9292ba57: Deleting consensus metadata
I20260927 02:41:59.958148 16545 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260927 02:41:59.958210 16545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.000s
I20260927 02:41:59.958289 16545 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:41:59.958313 16545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:41:59.966921 16545 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35573
I20260927 02:41:59.966961 16679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35573 every 8 connection(s)
I20260927 02:41:59.967787 16545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:41:59.976701 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 16545
Sep 27 02:41:59 dist-test-slave-zv8z krb5kdc[15772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:41:59.977090 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37805
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42269
--webserver_port=36161
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:41:59.981173 16543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:59.969504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57231 (local address 127.0.98.62:42269)
0927 02:41:59.969664 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 02:41:59.969669 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:41:59.970567 (+ 898us) server_negotiation.cc:407] Connection header received
0927 02:41:59.971501 (+ 934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:41:59.971504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:41:59.971555 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:41:59.971630 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:41:59.972794 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:59.973390 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:59.974545 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:41:59.974766 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:41:59.974845 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:59.977700 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:41:59.977703 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:41:59.977706 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:41:59.977737 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:41:59.979152 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:59.979740 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:59.979743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:59.979744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:59.979811 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:41:59.980129 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:41:59.980131 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:41:59.980132 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:41:59.980286 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:41:59.980354 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 02:41:59.980941 (+ 587us) server_negotiation.cc:299] Negotiation successful
0927 02:41:59.981055 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 02:41:59.982419 16682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:41:59.969942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.2:57231)
0927 02:41:59.970423 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0927 02:41:59.970466 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 02:41:59.971306 (+ 840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:41:59.971821 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:41:59.971829 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:41:59.972176 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:41:59.972642 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:59.972655 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:59.973516 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:59.973519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:59.974443 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:41:59.974452 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:41:59.974890 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:41:59.974893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:41:59.975091 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:41:59.975618 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:41:59.975634 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:41:59.977543 (+ 1909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:41:59.979263 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:59.979270 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:59.979285 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:59.979619 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:59.979884 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:41:59.979887 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:41:59.979889 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:41:59.980055 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:41:59.980373 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:41:59.980379 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:41:59.980694 (+ 315us) client_negotiation.cc:769] Sending connection context
0927 02:41:59.980907 (+ 213us) client_negotiation.cc:240] Negotiation successful
0927 02:41:59.981241 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":113,"threads_started":1}
I20260927 02:41:59.983635 16680 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42269
I20260927 02:41:59.983899 16680 heartbeater.cc:464] Registering TS with master...
I20260927 02:41:59.984472 16680 heartbeater.cc:511] Master 127.0.98.62:42269 requested a full tablet report, sending...
I20260927 02:41:59.985600 16353 ts_manager.cc:194] Registered new tserver with Master: e9a7c9d995ee4f51b1f1c8fe9292ba57 (127.0.98.2:35573)
I20260927 02:41:59.986217 16353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57231
W20260927 02:42:00.089474 16683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:00.089731 16683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:00.089774 16683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:00.093918 16683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:00.093984 16683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:00.094055 16683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:42:00.099289 16683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37805
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36161
--tserver_master_addrs=127.0.98.62:42269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:42:00.100289 16683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:00.101291 16683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:00.110509 16689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:00.110555 16683 server_base.cc:1034] running on GCE node
W20260927 02:42:00.110548 16688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:00.110502 16691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:00.111073 16683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:00.111636 16683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:00.112815 16683 hybrid_clock.cc:648] HybridClock initialized: now 1790476920112792 us; error 31 us; skew 500 ppm
Sep 27 02:42:00 dist-test-slave-zv8z krb5kdc[15772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:00.119737 16683 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:42:00.120730 16683 webserver.cc:460] Webserver started at http://127.0.98.3:36161/ using document root <none> and password file <none>
I20260927 02:42:00.121268 16683 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:00.121345 16683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:00.125138 16683 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:42:00.127175 16698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:00.128484 16683 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:42:00.128585 16683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-2/wal
uuid: "46370315edec4941bcd60a3c10b2b4a7"
format_stamp: "Formatted at 2026-09-27 02:41:39 on dist-test-slave-zv8z"
server_key: "dafdbd5c0ed0f5a156e001b95997a30c"
server_key_iv: "1a1c2589262cd937ca81ebd0b48ac77f"
server_key_version: "encryptionkey@0"
I20260927 02:42:00.129166 16683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:00.148321 16683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:00.152798 16683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:00.153189 16683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:00.153863 16683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:00.155236 16683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:00.155298 16683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:00.155366 16683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:00.155393 16683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:00.166008 16683 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37805
I20260927 02:42:00.166028 16811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37805 every 8 connection(s)
I20260927 02:42:00.167315 16683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:42:00.173698 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 16683
Sep 27 02:42:00 dist-test-slave-zv8z krb5kdc[15772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:42:00.182767 16543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:00.170465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49041 (local address 127.0.98.62:42269)
0927 02:42:00.170602 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:00.170608 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:00.170850 (+ 242us) server_negotiation.cc:407] Connection header received
0927 02:42:00.172008 (+ 1158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:00.172010 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:00.172055 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:00.172138 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:00.173711 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:00.174401 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:00.176092 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:00.176406 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:00.176493 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:00.179124 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:00.179128 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:00.179130 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:00.179155 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:00.180504 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:00.181088 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:00.181091 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:00.181092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:00.181166 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:00.181533 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:00.181537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:00.181539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:00.181763 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:00.181859 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:00.182490 (+ 631us) server_negotiation.cc:299] Negotiation successful
0927 02:42:00.182603 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 02:42:00.183595 16814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:00.169850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.3:49041)
0927 02:42:00.170684 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:00.170740 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:00.171794 (+ 1054us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:00.172356 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:00.172365 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:00.172830 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:00.173471 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:00.173490 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:00.174530 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:00.174534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:00.175968 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:00.175980 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:00.176551 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:00.176556 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:00.176761 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:00.177380 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:00.177397 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:00.178973 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:00.180605 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:00.180611 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:00.180627 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:00.180984 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:00.181251 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:00.181254 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:00.181256 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:00.181431 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:00.181914 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:00.181921 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:00.182211 (+ 290us) client_negotiation.cc:769] Sending connection context
0927 02:42:00.182438 (+ 227us) client_negotiation.cc:240] Negotiation successful
0927 02:42:00.182764 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":136,"threads_started":1}
I20260927 02:42:00.184237 16815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:00.174393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56790 (local address 127.0.98.62:42269)
0927 02:42:00.174641 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:00.174645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:00.174798 (+ 153us) server_negotiation.cc:407] Connection header received
0927 02:42:00.174904 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:00.174907 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:00.174950 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:00.175023 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:00.175744 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:00.176391 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:00.178282 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:00.178629 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:00.178736 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:00.180343 (+ 1607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:00.180347 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:00.180350 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:00.180391 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:00.182012 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:00.182551 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:00.182555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:00.182557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:00.182646 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:00.183235 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:00.183238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:00.183240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:00.183435 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:00.183641 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:00.183977 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 02:42:00.184085 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":80,"threads_started":1}
I20260927 02:42:00.184803 16812 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42269
I20260927 02:42:00.185025 16812 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:00.185545 16812 heartbeater.cc:511] Master 127.0.98.62:42269 requested a full tablet report, sending...
I20260927 02:42:00.186532 16353 ts_manager.cc:194] Registered new tserver with Master: 46370315edec4941bcd60a3c10b2b4a7 (127.0.98.3:37805)
I20260927 02:42:00.187067 16353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49041
I20260927 02:42:00.197628 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:42:00.201282 16815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:00.198201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56794 (local address 127.0.98.62:42269)
0927 02:42:00.198351 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:00.198354 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:00.198501 (+ 147us) server_negotiation.cc:407] Connection header received
0927 02:42:00.198626 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:00.198628 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:00.198733 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:00.198820 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:42:00.198966 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:00.199461 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:00.200286 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:00.200509 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:00.200584 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:00.200803 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:42:00.200939 (+ 136us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 02:42:00.201132 (+ 193us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56794: token expired
Metrics: {"server-negotiator.queue_time_us":70}
W20260927 02:42:00.201351 16815 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56794: token expired
W20260927 02:42:00.201735 16282 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:42269)
I20260927 02:42:00.209542 16815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:00.202064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56806 (local address 127.0.98.62:42269)
0927 02:42:00.202182 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:00.202185 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:00.202199 (+ 14us) server_negotiation.cc:407] Connection header received
0927 02:42:00.202233 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:00.202235 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:00.202356 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:00.202436 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:00.203083 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:00.203777 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:00.204562 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:00.204869 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:00.204962 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:00.206530 (+ 1568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:00.206534 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:00.206536 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:00.206564 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:00.208099 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:00.208464 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:00.208468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:00.208470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:00.208552 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:00.208765 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:00.208767 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:00.208768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:00.208929 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:00.209051 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:00.209288 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 02:42:00.209392 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 02:42:00.211829 16353 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56806:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:42:00.215142 16353 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:42:00.222966 16475 tablet_service.cc:1467] Processing CreateTablet for tablet 18ee83d58d224b038f82da4eaac1b961 (DEFAULT_TABLE table=test-table [id=bf6602fa02fa429bb140ff4b4939045d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:00.223620 16475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18ee83d58d224b038f82da4eaac1b961. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:00.226028 16433 tablet_bootstrap.cc:492] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed: Bootstrap starting.
I20260927 02:42:00.226955 16433 tablet_bootstrap.cc:654] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:00.227031 16822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:00.222009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35573 (local address 127.0.0.1:37146)
0927 02:42:00.222234 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:00.222271 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:00.222356 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:00.223054 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:00.223056 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:00.223073 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:00.223183 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:00.223190 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:00.224189 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:00.224192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:00.225518 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:00.225530 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:00.226570 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:00.226573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:00.226674 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:00.226722 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 02:42:00.226803 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 02:42:00.226877 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":19,"thread_start_us":87,"threads_started":1}
I20260927 02:42:00.227095 16823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:00.222282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37146 (local address 127.0.98.2:35573)
0927 02:42:00.222597 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:00.222608 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:00.222630 (+ 22us) server_negotiation.cc:407] Connection header received
0927 02:42:00.222729 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:00.222734 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:00.222943 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:00.223066 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:00.223286 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:00.224081 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:00.225656 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:00.226411 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:00.226554 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:00.226705 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:00.226839 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 02:42:00.226938 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":91,"threads_started":1}
I20260927 02:42:00.227741 16548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:00.221557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37805 (local address 127.0.0.1:56730)
0927 02:42:00.221670 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:00.221686 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:00.221762 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:00.222674 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:00.222677 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:00.222696 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:00.222794 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:00.222800 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:00.224302 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:00.224305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:00.225535 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:00.225541 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:00.227374 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:00.227378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:00.227471 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:00.227504 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 02:42:00.227553 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 02:42:00.227632 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":65}
I20260927 02:42:00.228334 16821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:00.221727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56730 (local address 127.0.98.3:37805)
0927 02:42:00.222109 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:00.222122 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:00.222145 (+ 23us) server_negotiation.cc:407] Connection header received
0927 02:42:00.222223 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:00.222228 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:00.222516 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:00.222686 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:00.222888 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:00.224191 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:00.225648 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:00.226944 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:00.227989 (+ 1045us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:00.228125 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:00.228164 (+ 39us) server_negotiation.cc:299] Negotiation successful
0927 02:42:00.228232 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":117,"threads_started":1}
I20260927 02:42:00.228344 16433 tablet_bootstrap.cc:492] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed: No bootstrap required, opened a new log
I20260927 02:42:00.228420 16433 ts_tablet_manager.cc:1397] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:42:00.228857 16433 raft_consensus.cc:348] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46370315edec4941bcd60a3c10b2b4a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37805 } } peers { permanent_uuid: "e9a7c9d995ee4f51b1f1c8fe9292ba57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35573 } } peers { permanent_uuid: "fca1d04e37bd41b4a5792c073f2808ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39877 } }
I20260927 02:42:00.228976 16433 raft_consensus.cc:374] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:00.229008 16433 raft_consensus.cc:729] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fca1d04e37bd41b4a5792c073f2808ed, State: Initialized, Role: FOLLOWER
I20260927 02:42:00.229130 16433 consensus_queue.cc:260] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46370315edec4941bcd60a3c10b2b4a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37805 } } peers { permanent_uuid: "e9a7c9d995ee4f51b1f1c8fe9292ba57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35573 } } peers { permanent_uuid: "fca1d04e37bd41b4a5792c073f2808ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39877 } }
I20260927 02:42:00.228717 16614 tablet_service.cc:1467] Processing CreateTablet for tablet 18ee83d58d224b038f82da4eaac1b961 (DEFAULT_TABLE table=test-table [id=bf6602fa02fa429bb140ff4b4939045d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:00.229861 16614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18ee83d58d224b038f82da4eaac1b961. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:00.230175 16433 ts_tablet_manager.cc:1428] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:42:00.232334 16746 tablet_service.cc:1467] Processing CreateTablet for tablet 18ee83d58d224b038f82da4eaac1b961 (DEFAULT_TABLE table=test-table [id=bf6602fa02fa429bb140ff4b4939045d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:00.233963 16746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18ee83d58d224b038f82da4eaac1b961. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:00.235352 16572 tablet_bootstrap.cc:492] T 18ee83d58d224b038f82da4eaac1b961 P e9a7c9d995ee4f51b1f1c8fe9292ba57: Bootstrap starting.
I20260927 02:42:00.237564 16572 tablet_bootstrap.cc:654] T 18ee83d58d224b038f82da4eaac1b961 P e9a7c9d995ee4f51b1f1c8fe9292ba57: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:00.238443 16572 log.cc:824] T 18ee83d58d224b038f82da4eaac1b961 P e9a7c9d995ee4f51b1f1c8fe9292ba57: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:00.240504 16572 tablet_bootstrap.cc:492] T 18ee83d58d224b038f82da4eaac1b961 P e9a7c9d995ee4f51b1f1c8fe9292ba57: No bootstrap required, opened a new log
I20260927 02:42:00.240690 16572 ts_tablet_manager.cc:1397] T 18ee83d58d224b038f82da4eaac1b961 P e9a7c9d995ee4f51b1f1c8fe9292ba57: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 02:42:00.241889 16826 tablet_bootstrap.cc:492] T 18ee83d58d224b038f82da4eaac1b961 P 46370315edec4941bcd60a3c10b2b4a7: Bootstrap starting.
I20260927 02:42:00.243783 16826 tablet_bootstrap.cc:654] T 18ee83d58d224b038f82da4eaac1b961 P 46370315edec4941bcd60a3c10b2b4a7: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:00.243881 16572 raft_consensus.cc:348] T 18ee83d58d224b038f82da4eaac1b961 P e9a7c9d995ee4f51b1f1c8fe9292ba57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46370315edec4941bcd60a3c10b2b4a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37805 } } peers { permanent_uuid: "e9a7c9d995ee4f51b1f1c8fe9292ba57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35573 } } peers { permanent_uuid: "fca1d04e37bd41b4a5792c073f2808ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39877 } }
I20260927 02:42:00.244100 16572 raft_consensus.cc:374] T 18ee83d58d224b038f82da4eaac1b961 P e9a7c9d995ee4f51b1f1c8fe9292ba57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:00.244164 16572 raft_consensus.cc:729] T 18ee83d58d224b038f82da4eaac1b961 P e9a7c9d995ee4f51b1f1c8fe9292ba57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9a7c9d995ee4f51b1f1c8fe9292ba57, State: Initialized, Role: FOLLOWER
I20260927 02:42:00.244545 16572 consensus_queue.cc:260] T 18ee83d58d224b038f82da4eaac1b961 P e9a7c9d995ee4f51b1f1c8fe9292ba57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46370315edec4941bcd60a3c10b2b4a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37805 } } peers { permanent_uuid: "e9a7c9d995ee4f51b1f1c8fe9292ba57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35573 } } peers { permanent_uuid: "fca1d04e37bd41b4a5792c073f2808ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39877 } }
I20260927 02:42:00.244833 16826 log.cc:824] T 18ee83d58d224b038f82da4eaac1b961 P 46370315edec4941bcd60a3c10b2b4a7: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:00.245138 16680 heartbeater.cc:503] Master 127.0.98.62:42269 was elected leader, sending a full tablet report...
I20260927 02:42:00.245471 16572 ts_tablet_manager.cc:1428] T 18ee83d58d224b038f82da4eaac1b961 P e9a7c9d995ee4f51b1f1c8fe9292ba57: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 02:42:00.246762 16826 tablet_bootstrap.cc:492] T 18ee83d58d224b038f82da4eaac1b961 P 46370315edec4941bcd60a3c10b2b4a7: No bootstrap required, opened a new log
I20260927 02:42:00.246934 16826 ts_tablet_manager.cc:1397] T 18ee83d58d224b038f82da4eaac1b961 P 46370315edec4941bcd60a3c10b2b4a7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 02:42:00.249765 16826 raft_consensus.cc:348] T 18ee83d58d224b038f82da4eaac1b961 P 46370315edec4941bcd60a3c10b2b4a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46370315edec4941bcd60a3c10b2b4a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37805 } } peers { permanent_uuid: "e9a7c9d995ee4f51b1f1c8fe9292ba57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35573 } } peers { permanent_uuid: "fca1d04e37bd41b4a5792c073f2808ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39877 } }
I20260927 02:42:00.249975 16826 raft_consensus.cc:374] T 18ee83d58d224b038f82da4eaac1b961 P 46370315edec4941bcd60a3c10b2b4a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:00.250036 16826 raft_consensus.cc:729] T 18ee83d58d224b038f82da4eaac1b961 P 46370315edec4941bcd60a3c10b2b4a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46370315edec4941bcd60a3c10b2b4a7, State: Initialized, Role: FOLLOWER
I20260927 02:42:00.250442 16826 consensus_queue.cc:260] T 18ee83d58d224b038f82da4eaac1b961 P 46370315edec4941bcd60a3c10b2b4a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46370315edec4941bcd60a3c10b2b4a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37805 } } peers { permanent_uuid: "e9a7c9d995ee4f51b1f1c8fe9292ba57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35573 } } peers { permanent_uuid: "fca1d04e37bd41b4a5792c073f2808ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39877 } }
I20260927 02:42:00.250996 16812 heartbeater.cc:503] Master 127.0.98.62:42269 was elected leader, sending a full tablet report...
I20260927 02:42:00.251217 16826 ts_tablet_manager.cc:1428] T 18ee83d58d224b038f82da4eaac1b961 P 46370315edec4941bcd60a3c10b2b4a7: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20260927 02:42:00.279019 16542 tablet.cc:2369] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:42:00.418305 16813 tablet.cc:2369] T 18ee83d58d224b038f82da4eaac1b961 P 46370315edec4941bcd60a3c10b2b4a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:42:00.468883 16681 tablet.cc:2369] T 18ee83d58d224b038f82da4eaac1b961 P e9a7c9d995ee4f51b1f1c8fe9292ba57: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:42:00.491485 16827 raft_consensus.cc:482] T 18ee83d58d224b038f82da4eaac1b961 P e9a7c9d995ee4f51b1f1c8fe9292ba57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:42:00.491734 16827 raft_consensus.cc:504] T 18ee83d58d224b038f82da4eaac1b961 P e9a7c9d995ee4f51b1f1c8fe9292ba57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46370315edec4941bcd60a3c10b2b4a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37805 } } peers { permanent_uuid: "e9a7c9d995ee4f51b1f1c8fe9292ba57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35573 } } peers { permanent_uuid: "fca1d04e37bd41b4a5792c073f2808ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39877 } }
I20260927 02:42:00.492847 16827 leader_election.cc:288] T 18ee83d58d224b038f82da4eaac1b961 P e9a7c9d995ee4f51b1f1c8fe9292ba57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46370315edec4941bcd60a3c10b2b4a7 (127.0.98.3:37805), fca1d04e37bd41b4a5792c073f2808ed (127.0.98.1:39877)
I20260927 02:42:00.496551 16821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:00.493468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35693 (local address 127.0.98.3:37805)
0927 02:42:00.493591 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:00.493596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:00.493630 (+ 34us) server_negotiation.cc:407] Connection header received
0927 02:42:00.493673 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:00.493675 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:00.493714 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:00.493775 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:00.494053 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:00.494621 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:00.495648 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:00.496060 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:00.496149 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:00.496210 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:00.496367 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 02:42:00.496434 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 02:42:00.496567 16830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:00.493088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37805 (local address 127.0.98.2:35693)
0927 02:42:00.493346 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:00.493361 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:00.493479 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:00.493798 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:00.493801 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:00.493816 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:00.493948 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:00.493953 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:00.494716 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:00.494719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:00.495522 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:00.495530 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:00.496165 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:00.496169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:00.496258 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:00.496276 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 02:42:00.496339 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 02:42:00.496416 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":23,"thread_start_us":95,"threads_started":1}
I20260927 02:42:00.496641 16832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:00.493227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38619 (local address 127.0.98.1:39877)
0927 02:42:00.493451 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:00.493457 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:00.493521 (+ 64us) server_negotiation.cc:407] Connection header received
0927 02:42:00.493666 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:00.493669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:00.493723 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:00.493810 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:00.494135 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:00.494683 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:00.495680 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:00.496121 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:00.496214 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:00.496374 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:00.496438 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 02:42:00.496502 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":64,"threads_started":1}
I20260927 02:42:00.496690 16831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:00.493129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39877 (local address 127.0.98.2:38619)
0927 02:42:00.493442 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:00.493456 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:00.493566 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:00.493835 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:00.493838 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:00.493852 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:00.493981 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:00.493986 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:00.494785 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:00.494788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:00.495567 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:00.495575 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:00.496233 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:00.496237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:00.496336 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:00.496356 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 02:42:00.496428 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 02:42:00.496495 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":1,"thread_start_us":107,"threads_started":1}
I20260927 02:42:00.497505 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18ee83d58d224b038f82da4eaac1b961" candidate_uuid: "e9a7c9d995ee4f51b1f1c8fe9292ba57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fca1d04e37bd41b4a5792c073f2808ed" is_pre_election: true
I20260927 02:42:00.497556 16766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18ee83d58d224b038f82da4eaac1b961" candidate_uuid: "e9a7c9d995ee4f51b1f1c8fe9292ba57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46370315edec4941bcd60a3c10b2b4a7" is_pre_election: true
I20260927 02:42:00.497778 16495 raft_consensus.cc:2454] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9a7c9d995ee4f51b1f1c8fe9292ba57 in term 0.
I20260927 02:42:00.497816 16766 raft_consensus.cc:2454] T 18ee83d58d224b038f82da4eaac1b961 P 46370315edec4941bcd60a3c10b2b4a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9a7c9d995ee4f51b1f1c8fe9292ba57 in term 0.
I20260927 02:42:00.498327 16568 leader_election.cc:302] T 18ee83d58d224b038f82da4eaac1b961 P e9a7c9d995ee4f51b1f1c8fe9292ba57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e9a7c9d995ee4f51b1f1c8fe9292ba57, fca1d04e37bd41b4a5792c073f2808ed; no voters:
I20260927 02:42:00.498584 16827 raft_consensus.cc:2781] T 18ee83d58d224b038f82da4eaac1b961 P e9a7c9d995ee4f51b1f1c8fe9292ba57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:42:00.498667 16827 raft_consensus.cc:482] T 18ee83d58d224b038f82da4eaac1b961 P e9a7c9d995ee4f51b1f1c8fe9292ba57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:42:00.498699 16827 raft_consensus.cc:3037] T 18ee83d58d224b038f82da4eaac1b961 P e9a7c9d995ee4f51b1f1c8fe9292ba57 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:00.499717 16827 raft_consensus.cc:504] T 18ee83d58d224b038f82da4eaac1b961 P e9a7c9d995ee4f51b1f1c8fe9292ba57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46370315edec4941bcd60a3c10b2b4a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37805 } } peers { permanent_uuid: "e9a7c9d995ee4f51b1f1c8fe9292ba57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35573 } } peers { permanent_uuid: "fca1d04e37bd41b4a5792c073f2808ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39877 } }
I20260927 02:42:00.500047 16827 leader_election.cc:288] T 18ee83d58d224b038f82da4eaac1b961 P e9a7c9d995ee4f51b1f1c8fe9292ba57 [CANDIDATE]: Term 1 election: Requested vote from peers 46370315edec4941bcd60a3c10b2b4a7 (127.0.98.3:37805), fca1d04e37bd41b4a5792c073f2808ed (127.0.98.1:39877)
I20260927 02:42:00.500375 16766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18ee83d58d224b038f82da4eaac1b961" candidate_uuid: "e9a7c9d995ee4f51b1f1c8fe9292ba57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46370315edec4941bcd60a3c10b2b4a7"
I20260927 02:42:00.500391 16495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18ee83d58d224b038f82da4eaac1b961" candidate_uuid: "e9a7c9d995ee4f51b1f1c8fe9292ba57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fca1d04e37bd41b4a5792c073f2808ed"
I20260927 02:42:00.500499 16495 raft_consensus.cc:3037] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:00.500500 16766 raft_consensus.cc:3037] T 18ee83d58d224b038f82da4eaac1b961 P 46370315edec4941bcd60a3c10b2b4a7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:00.501646 16766 raft_consensus.cc:2454] T 18ee83d58d224b038f82da4eaac1b961 P 46370315edec4941bcd60a3c10b2b4a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9a7c9d995ee4f51b1f1c8fe9292ba57 in term 1.
I20260927 02:42:00.501657 16495 raft_consensus.cc:2454] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9a7c9d995ee4f51b1f1c8fe9292ba57 in term 1.
I20260927 02:42:00.501945 16566 leader_election.cc:302] T 18ee83d58d224b038f82da4eaac1b961 P e9a7c9d995ee4f51b1f1c8fe9292ba57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46370315edec4941bcd60a3c10b2b4a7, e9a7c9d995ee4f51b1f1c8fe9292ba57; no voters:
I20260927 02:42:00.502100 16827 raft_consensus.cc:2781] T 18ee83d58d224b038f82da4eaac1b961 P e9a7c9d995ee4f51b1f1c8fe9292ba57 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:00.502344 16827 raft_consensus.cc:686] T 18ee83d58d224b038f82da4eaac1b961 P e9a7c9d995ee4f51b1f1c8fe9292ba57 [term 1 LEADER]: Becoming Leader. State: Replica: e9a7c9d995ee4f51b1f1c8fe9292ba57, State: Running, Role: LEADER
I20260927 02:42:00.502677 16827 consensus_queue.cc:237] T 18ee83d58d224b038f82da4eaac1b961 P e9a7c9d995ee4f51b1f1c8fe9292ba57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46370315edec4941bcd60a3c10b2b4a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37805 } } peers { permanent_uuid: "e9a7c9d995ee4f51b1f1c8fe9292ba57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35573 } } peers { permanent_uuid: "fca1d04e37bd41b4a5792c073f2808ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39877 } }
I20260927 02:42:00.505712 16353 catalog_manager.cc:5495] T 18ee83d58d224b038f82da4eaac1b961 P e9a7c9d995ee4f51b1f1c8fe9292ba57 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9a7c9d995ee4f51b1f1c8fe9292ba57 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "e9a7c9d995ee4f51b1f1c8fe9292ba57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46370315edec4941bcd60a3c10b2b4a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9a7c9d995ee4f51b1f1c8fe9292ba57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35573 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fca1d04e37bd41b4a5792c073f2808ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39877 } health_report { overall_health: UNKNOWN } } }
I20260927 02:42:00.526028 16823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:00.523005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37162 (local address 127.0.98.2:35573)
0927 02:42:00.523170 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:00.523174 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:00.523187 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:42:00.523289 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:00.523292 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:00.523343 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:00.523427 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:42:00.523580 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:00.524059 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:00.524829 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:00.525052 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:00.525136 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:00.525366 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:42:00.525767 (+ 401us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:42:00.525850 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:00.525877 (+ 27us) server_negotiation.cc:299] Negotiation successful
0927 02:42:00.525923 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260927 02:42:00.531862 16495 raft_consensus.cc:1261] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed [term 1 FOLLOWER]: Refusing update from remote peer e9a7c9d995ee4f51b1f1c8fe9292ba57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:42:00.531867 16766 raft_consensus.cc:1261] T 18ee83d58d224b038f82da4eaac1b961 P 46370315edec4941bcd60a3c10b2b4a7 [term 1 FOLLOWER]: Refusing update from remote peer e9a7c9d995ee4f51b1f1c8fe9292ba57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:42:00.532541 16833 consensus_queue.cc:1035] T 18ee83d58d224b038f82da4eaac1b961 P e9a7c9d995ee4f51b1f1c8fe9292ba57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46370315edec4941bcd60a3c10b2b4a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:42:00.532703 16827 consensus_queue.cc:1035] T 18ee83d58d224b038f82da4eaac1b961 P e9a7c9d995ee4f51b1f1c8fe9292ba57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fca1d04e37bd41b4a5792c073f2808ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:42:00.538694 16836 mvcc.cc:205] Tried to move back new op lower bound from 7333793466491281408 to 7333793466381377536. Current Snapshot: MvccSnapshot[applied={T|T < 7333793466491281408}]
I20260927 02:42:00.539320 16838 mvcc.cc:205] Tried to move back new op lower bound from 7333793466491281408 to 7333793466381377536. Current Snapshot: MvccSnapshot[applied={T|T < 7333793466491281408}]
I20260927 02:42:00.547098 16351 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:42:00.547247 16351 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:42:00.549472 16351 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4: Sending DeleteTablet for 3 replicas of tablet 18ee83d58d224b038f82da4eaac1b961
I20260927 02:42:00.550098 16746 tablet_service.cc:1514] Processing DeleteTablet for tablet 18ee83d58d224b038f82da4eaac1b961 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56730
I20260927 02:42:00.550362 16614 tablet_service.cc:1514] Processing DeleteTablet for tablet 18ee83d58d224b038f82da4eaac1b961 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37146
I20260927 02:42:00.550791 16846 tablet_replica.cc:330] T 18ee83d58d224b038f82da4eaac1b961 P 46370315edec4941bcd60a3c10b2b4a7: stopping tablet replica
I20260927 02:42:00.551151 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 16391
I20260927 02:42:00.551152 16847 tablet_replica.cc:330] T 18ee83d58d224b038f82da4eaac1b961 P e9a7c9d995ee4f51b1f1c8fe9292ba57: stopping tablet replica
I20260927 02:42:00.551424 16847 raft_consensus.cc:2229] T 18ee83d58d224b038f82da4eaac1b961 P e9a7c9d995ee4f51b1f1c8fe9292ba57 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:42:00.551848 16847 raft_consensus.cc:2258] T 18ee83d58d224b038f82da4eaac1b961 P e9a7c9d995ee4f51b1f1c8fe9292ba57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:00.555420 16847 ts_tablet_manager.cc:1905] T 18ee83d58d224b038f82da4eaac1b961 P e9a7c9d995ee4f51b1f1c8fe9292ba57: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 02:42:00.557845 16339 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39877 (error 108)
I20260927 02:42:00.558048 16847 ts_tablet_manager.cc:1918] T 18ee83d58d224b038f82da4eaac1b961 P e9a7c9d995ee4f51b1f1c8fe9292ba57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:42:00.558109 16847 log.cc:1196] T 18ee83d58d224b038f82da4eaac1b961 P e9a7c9d995ee4f51b1f1c8fe9292ba57: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-1/wal/wals/18ee83d58d224b038f82da4eaac1b961
I20260927 02:42:00.558456 16847 ts_tablet_manager.cc:1939] T 18ee83d58d224b038f82da4eaac1b961 P e9a7c9d995ee4f51b1f1c8fe9292ba57: Deleting consensus metadata
I20260927 02:42:00.558966 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 16545
I20260927 02:42:00.560374 16846 raft_consensus.cc:2229] T 18ee83d58d224b038f82da4eaac1b961 P 46370315edec4941bcd60a3c10b2b4a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:42:00.560706 16846 raft_consensus.cc:2258] T 18ee83d58d224b038f82da4eaac1b961 P 46370315edec4941bcd60a3c10b2b4a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:00.561692 16548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:00.561363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39877 (local address 127.0.0.1:57416)
0927 02:42:00.561550 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:00.561618 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260927 02:42:00.561916 16339 catalog_manager.cc:4584] TS fca1d04e37bd41b4a5792c073f2808ed (127.0.98.1:39877): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 18ee83d58d224b038f82da4eaac1b961: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39877: connect: Connection refused (error 111)
I20260927 02:42:00.562510 16846 ts_tablet_manager.cc:1905] T 18ee83d58d224b038f82da4eaac1b961 P 46370315edec4941bcd60a3c10b2b4a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:42:00.565018 16846 ts_tablet_manager.cc:1918] T 18ee83d58d224b038f82da4eaac1b961 P 46370315edec4941bcd60a3c10b2b4a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:42:00.565111 16846 log.cc:1196] T 18ee83d58d224b038f82da4eaac1b961 P 46370315edec4941bcd60a3c10b2b4a7: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-2/wal/wals/18ee83d58d224b038f82da4eaac1b961
I20260927 02:42:00.565411 16846 ts_tablet_manager.cc:1939] T 18ee83d58d224b038f82da4eaac1b961 P 46370315edec4941bcd60a3c10b2b4a7: Deleting consensus metadata
I20260927 02:42:00.566640 16337 catalog_manager.cc:4841] TS 46370315edec4941bcd60a3c10b2b4a7 (127.0.98.3:37805): tablet 18ee83d58d224b038f82da4eaac1b961 (table test-table [id=bf6602fa02fa429bb140ff4b4939045d]) successfully deleted
I20260927 02:42:00.568368 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 16683
I20260927 02:42:00.569303 16548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:00.569015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35573 (local address 127.0.0.1:37168)
0927 02:42:00.569173 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:00.569227 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260927 02:42:00.573149 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 16320
I20260927 02:42:20.578397 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42269
--webserver_interface=127.0.98.62
--webserver_port=41303
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:20.682945 16849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:20.683203 16849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:20.683247 16849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:20.687183 16849 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 02:42:20.687237 16849 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 02:42:20.687250 16849 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 02:42:20.687263 16849 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:42:20.687279 16849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:20.687297 16849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:20.687333 16849 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:42:20.687359 16849 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:42:20.692517 16849 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:42:20.693539 16849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:20.694589 16849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:20.702708 16854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:20.702719 16855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:20.702862 16849 server_base.cc:1034] running on GCE node
W20260927 02:42:20.702726 16857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:20.703552 16849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:20.704618 16849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:20.705786 16849 hybrid_clock.cc:648] HybridClock initialized: now 1790476940705772 us; error 34 us; skew 500 ppm
Sep 27 02:42:20 dist-test-slave-zv8z krb5kdc[15772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476940, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:20.708585 16849 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:42:20.709434 16849 webserver.cc:460] Webserver started at http://127.0.98.62:41303/ using document root <none> and password file <none>
I20260927 02:42:20.709955 16849 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:20.710021 16849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:20.713682 16849 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260927 02:42:20.715802 16864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:20.717097 16849 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:42:20.717213 16849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "57522849be664587a9597a5f173c4fd4"
format_stamp: "Formatted at 2026-09-27 02:41:38 on dist-test-slave-zv8z"
server_key: "abbb70c17717280875fec0f01accbeba"
server_key_iv: "317f0961e59f624e9656eae615c27ab3"
server_key_version: "encryptionkey@0"
I20260927 02:42:20.717748 16849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:20.750626 16849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:20.753578 16849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:20.753757 16849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:20.760437 16849 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42269
I20260927 02:42:20.760695 16916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42269 every 8 connection(s)
I20260927 02:42:20.761461 16849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:42:20.764514 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 16849
I20260927 02:42:20.764976 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39877
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42269
--webserver_port=33605
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:42:20.765990 16917 sys_catalog.cc:263] Verifying existing consensus state
I20260927 02:42:20.767675 16917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4: Bootstrap starting.
I20260927 02:42:20.784505 16917 log.cc:824] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:20.795725 16917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:42:20.796054 16917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4: Bootstrap complete.
I20260927 02:42:20.799899 16917 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57522849be664587a9597a5f173c4fd4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42269 } }
I20260927 02:42:20.800292 16917 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57522849be664587a9597a5f173c4fd4, State: Initialized, Role: FOLLOWER
I20260927 02:42:20.800752 16917 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57522849be664587a9597a5f173c4fd4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42269 } }
I20260927 02:42:20.800863 16917 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:42:20.800925 16917 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:42:20.801014 16917 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4 [term 2 FOLLOWER]: Advancing to term 3
I20260927 02:42:20.802832 16917 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57522849be664587a9597a5f173c4fd4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42269 } }
I20260927 02:42:20.803133 16917 leader_election.cc:302] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57522849be664587a9597a5f173c4fd4; no voters:
I20260927 02:42:20.803437 16917 leader_election.cc:288] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 02:42:20.803509 16921 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 02:42:20.803697 16921 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4 [term 3 LEADER]: Becoming Leader. State: Replica: 57522849be664587a9597a5f173c4fd4, State: Running, Role: LEADER
I20260927 02:42:20.804034 16921 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57522849be664587a9597a5f173c4fd4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42269 } }
I20260927 02:42:20.804608 16917 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:42:20.805019 16921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57522849be664587a9597a5f173c4fd4. Latest consensus state: current_term: 3 leader_uuid: "57522849be664587a9597a5f173c4fd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57522849be664587a9597a5f173c4fd4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42269 } } }
I20260927 02:42:20.805127 16921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:20.805476 16922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "57522849be664587a9597a5f173c4fd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57522849be664587a9597a5f173c4fd4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42269 } } }
I20260927 02:42:20.805568 16922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:20.806407 16930 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:42:20.811369 16930 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:42:20.811944 16930 catalog_manager.cc:1259] Loaded cluster ID: 967a46d087bf4409992cdc59e4368267
I20260927 02:42:20.812001 16930 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:42:20.814363 16930 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:42:20.815092 16930 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4: Loaded TSK: 0
I20260927 02:42:20.815179 16930 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 57522849be664587a9597a5f173c4fd4: Loaded TSK: 1
F20260927 02:42:20.815850 16930 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f253adaa9ad google::LogMessage::Fail() at ??:0
@ 0x7f253adad88a google::LogMessage::SendToLog() at ??:0
@ 0x7f253adaa4de google::LogMessage::Flush() at ??:0
@ 0x7f253adaba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f253da6e18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f253da66f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f254570f7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f25456e999c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f25456ea178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f254572bdf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f254574a82b std::function<>::operator()() at ??:0
@ 0x7f25456e9b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f25456eaf3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f25456e7579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f254572b640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f253cffc3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f253cffcc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f253cffe3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f253cfed512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f253e6316db start_thread at ??:0
@ 0x7f25398a861f clone at ??:0
W20260927 02:42:20.903613 16919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:20.903951 16919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:20.904017 16919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:20.908664 16919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:20.908723 16919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:20.908789 16919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:42:20.913902 16919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39877
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33605
--tserver_master_addrs=127.0.98.62:42269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:42:20.914911 16919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:20.915887 16919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:20.924952 16964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:20.925017 16965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:20.924947 16967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:20.925585 16919 server_base.cc:1034] running on GCE node
I20260927 02:42:20.925961 16919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:20.926568 16919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:20.927723 16919 hybrid_clock.cc:648] HybridClock initialized: now 1790476940927693 us; error 49 us; skew 500 ppm
Sep 27 02:42:20 dist-test-slave-zv8z krb5kdc[15772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476940, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:20.930568 16919 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:42:20.931450 16919 webserver.cc:460] Webserver started at http://127.0.98.1:33605/ using document root <none> and password file <none>
I20260927 02:42:20.931965 16919 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:20.932030 16919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:20.935680 16919 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.002s
I20260927 02:42:20.937696 16974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:20.938853 16919 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:42:20.938942 16919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-0/wal
uuid: "fca1d04e37bd41b4a5792c073f2808ed"
format_stamp: "Formatted at 2026-09-27 02:41:38 on dist-test-slave-zv8z"
server_key: "e94d55e37f959a06487a904c344a2a01"
server_key_iv: "5667ca9bccac7daea5a2c57cfd61c9eb"
server_key_version: "encryptionkey@0"
I20260927 02:42:20.939458 16919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:20.952999 16919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:20.955992 16919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:20.956153 16919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:20.956682 16919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:20.957710 16981 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 02:42:20.962931 16919 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 02:42:20.962991 16919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260927 02:42:20.963049 16919 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 02:42:20.964623 16919 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 02:42:20.964675 16919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 02:42:20.964972 16981 tablet_bootstrap.cc:492] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed: Bootstrap starting.
I20260927 02:42:20.973449 16919 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39877
I20260927 02:42:20.973694 17088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39877 every 8 connection(s)
I20260927 02:42:20.974517 16919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:42:20.981437 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 16919
I20260927 02:42:20.981845 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35573
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42269
--webserver_port=34579
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:42:20.988092 16981 log.cc:824] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:20.994944 16981 tablet_bootstrap.cc:492] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:42:20.995286 16981 tablet_bootstrap.cc:492] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed: Bootstrap complete.
I20260927 02:42:20.996205 16981 ts_tablet_manager.cc:1397] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed: Time spent bootstrapping tablet: real 0.031s user 0.020s sys 0.007s
I20260927 02:42:20.998693 16981 raft_consensus.cc:348] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46370315edec4941bcd60a3c10b2b4a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37805 } } peers { permanent_uuid: "e9a7c9d995ee4f51b1f1c8fe9292ba57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35573 } } peers { permanent_uuid: "fca1d04e37bd41b4a5792c073f2808ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39877 } }
I20260927 02:42:20.999121 16981 raft_consensus.cc:729] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fca1d04e37bd41b4a5792c073f2808ed, State: Initialized, Role: FOLLOWER
I20260927 02:42:20.999610 16981 consensus_queue.cc:260] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46370315edec4941bcd60a3c10b2b4a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37805 } } peers { permanent_uuid: "e9a7c9d995ee4f51b1f1c8fe9292ba57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35573 } } peers { permanent_uuid: "fca1d04e37bd41b4a5792c073f2808ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39877 } }
I20260927 02:42:21.000545 16981 ts_tablet_manager.cc:1428] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20260927 02:42:21.039935 17091 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:42:20.976359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.1:46903)
0927 02:42:20.976827 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:20.976880 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:20.977683 (+ 803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:21.039216 (+ 61533us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42269: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":111,"threads_started":1}
W20260927 02:42:21.040594 17089 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42269 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:42269: Client connection negotiation failed: client connection to 127.0.98.62:42269: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 02:42:21.041370 17091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:21.041059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.1:42051)
0927 02:42:21.041209 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:21.041296 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 02:42:21.042017 17091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:21.041788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.1:39169)
0927 02:42:21.041900 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:21.041955 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 02:42:21.042179 17089 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 02:42:21.109301 17092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:21.109565 17092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:21.109611 17092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:21.113574 17092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:21.113629 17092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:21.113685 17092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:42:21.118772 17092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35573
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34579
--tserver_master_addrs=127.0.98.62:42269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:42:21.119722 17092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:21.120735 17092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:21.129545 17100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:21.129601 17099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:21.129705 17092 server_base.cc:1034] running on GCE node
W20260927 02:42:21.129557 17102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:21.130318 17092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:21.130918 17092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:21.132143 17092 hybrid_clock.cc:648] HybridClock initialized: now 1790476941132095 us; error 57 us; skew 500 ppm
Sep 27 02:42:21 dist-test-slave-zv8z krb5kdc[15772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476941, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:21.135109 17092 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:42:21.135998 17092 webserver.cc:460] Webserver started at http://127.0.98.2:34579/ using document root <none> and password file <none>
I20260927 02:42:21.136536 17092 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:21.136613 17092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:21.140496 17092 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:42:21.142647 17109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:21.144109 17092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:42:21.144233 17092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-1/wal
uuid: "e9a7c9d995ee4f51b1f1c8fe9292ba57"
format_stamp: "Formatted at 2026-09-27 02:41:38 on dist-test-slave-zv8z"
server_key: "5353c03c6d9552a836be848f6241225f"
server_key_iv: "a25653b8b69b5d81c5f5b950f8420ded"
server_key_version: "encryptionkey@0"
I20260927 02:42:21.144786 17092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:21.161300 17092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:21.164285 17092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:21.164459 17092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:21.165017 17092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:21.165812 17092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:21.165860 17092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:21.165910 17092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:21.165939 17092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:21.174207 17092 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35573
I20260927 02:42:21.174278 17222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35573 every 8 connection(s)
I20260927 02:42:21.175225 17092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:42:21.179114 17225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:21.177365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.2:34123)
0927 02:42:21.178295 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:21.178421 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":667,"thread_start_us":504,"threads_started":1}
W20260927 02:42:21.179589 17223 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42269 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:42269: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
I20260927 02:42:21.180194 17225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:21.179967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.2:60241)
0927 02:42:21.180094 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:21.180141 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 02:42:21.180739 17225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:21.180555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.2:35007)
0927 02:42:21.180649 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:21.180689 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 02:42:21.180924 17223 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 02:42:21.180991 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 17092
I20260927 02:42:21.181353 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37805
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42269
--webserver_port=36161
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:21.288568 17226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:21.288813 17226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:21.288852 17226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:21.292745 17226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:21.292799 17226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:21.292853 17226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:42:21.297863 17226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37805
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36161
--tserver_master_addrs=127.0.98.62:42269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:42:21.298870 17226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:21.299815 17226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:21.309412 17232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:21.309486 17231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:21.309598 17226 server_base.cc:1034] running on GCE node
W20260927 02:42:21.309494 17234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:21.310047 17226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:21.310729 17226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:21.311890 17226 hybrid_clock.cc:648] HybridClock initialized: now 1790476941311875 us; error 29 us; skew 500 ppm
Sep 27 02:42:21 dist-test-slave-zv8z krb5kdc[15772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476941, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:21.314468 17226 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:42:21.315297 17226 webserver.cc:460] Webserver started at http://127.0.98.3:36161/ using document root <none> and password file <none>
I20260927 02:42:21.315799 17226 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:21.315855 17226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:21.319638 17226 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260927 02:42:21.321789 17241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:21.323002 17226 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:42:21.323098 17226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-2/wal
uuid: "46370315edec4941bcd60a3c10b2b4a7"
format_stamp: "Formatted at 2026-09-27 02:41:39 on dist-test-slave-zv8z"
server_key: "dafdbd5c0ed0f5a156e001b95997a30c"
server_key_iv: "1a1c2589262cd937ca81ebd0b48ac77f"
server_key_version: "encryptionkey@0"
I20260927 02:42:21.323619 17226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:21.340478 17226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:21.343703 17226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:21.343875 17226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:21.344465 17226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:21.345252 17226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:21.345309 17226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:21.345346 17226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:21.345363 17226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:21.354089 17226 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37805
I20260927 02:42:21.354132 17354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37805 every 8 connection(s)
I20260927 02:42:21.355096 17226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:42:21.357165 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 17226
I20260927 02:42:21.358809 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:21.357157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.3:39025)
0927 02:42:21.357857 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:21.357992 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":124,"threads_started":1}
W20260927 02:42:21.359196 17355 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42269 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:42269: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
I20260927 02:42:21.359746 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:21.359528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.3:50543)
0927 02:42:21.359652 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:21.359695 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 02:42:21.360291 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:21.360106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.3:54247)
0927 02:42:21.360196 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:21.360240 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 02:42:21.360424 17355 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 02:42:22.043066 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:22.042686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.1:58071)
0927 02:42:22.042898 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:22.042954 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":67,"threads_started":1}
I20260927 02:42:22.181773 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:22.181413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.2:53969)
0927 02:42:22.181623 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:22.181672 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":63,"threads_started":1}
I20260927 02:42:22.361374 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:22.360968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.3:36433)
0927 02:42:22.361215 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:22.361276 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":76,"threads_started":1}
I20260927 02:42:22.818688 17362 raft_consensus.cc:482] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:42:22.818943 17362 raft_consensus.cc:504] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46370315edec4941bcd60a3c10b2b4a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37805 } } peers { permanent_uuid: "e9a7c9d995ee4f51b1f1c8fe9292ba57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35573 } } peers { permanent_uuid: "fca1d04e37bd41b4a5792c073f2808ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39877 } }
I20260927 02:42:22.820084 17362 leader_election.cc:288] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46370315edec4941bcd60a3c10b2b4a7 (127.0.98.3:37805), e9a7c9d995ee4f51b1f1c8fe9292ba57 (127.0.98.2:35573)
Sep 27 02:42:22 dist-test-slave-zv8z krb5kdc[15772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476940, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 27 02:42:22 dist-test-slave-zv8z krb5kdc[15772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476940, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260927 02:42:22.833174 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:22.820447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45353 (local address 127.0.98.3:37805)
0927 02:42:22.820752 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:22.820780 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:22.821096 (+ 316us) server_negotiation.cc:407] Connection header received
0927 02:42:22.821786 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:22.821791 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:22.822071 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:22.822392 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:22.823559 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:22.824496 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:22.825638 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:22.826078 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:22.826235 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:22.828870 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:22.828875 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:22.828886 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:22.828913 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:22.830764 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:22.831352 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:22.831358 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:22.831363 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:22.831471 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:22.831779 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:22.831782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:22.831784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:22.832158 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:22.832270 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:22.832763 (+ 493us) server_negotiation.cc:299] Negotiation successful
0927 02:42:22.833009 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":95,"threads_started":1}
I20260927 02:42:22.833173 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:22.820313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37805 (local address 127.0.98.1:45353)
0927 02:42:22.820968 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:22.820981 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:22.821101 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:22.822368 (+ 1267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:22.822374 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:22.822892 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:22.823401 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:22.823416 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:22.824628 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:22.824630 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:22.825529 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:22.825538 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:22.826198 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:22.826202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:22.826452 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:22.827131 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:22.827142 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:22.828632 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:22.830874 (+ 2242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:22.830880 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:22.830893 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:22.831248 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:22.831556 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:22.831559 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:22.831560 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:22.831692 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:22.832263 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:22.832269 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:22.832475 (+ 206us) client_negotiation.cc:769] Sending connection context
0927 02:42:22.832639 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 02:42:22.832969 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"mutex_wait_us":528,"spinlock_wait_cycles":5760,"thread_start_us":71,"threads_started":1}
I20260927 02:42:22.833210 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:22.820313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35573 (local address 127.0.98.1:35073)
0927 02:42:22.820644 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:22.820659 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:22.820766 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:22.822369 (+ 1603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:22.822374 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:22.822893 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:22.823415 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:22.823424 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:22.824630 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:22.824632 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:22.825528 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:22.825537 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:22.826199 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:22.826203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:22.826455 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:22.827016 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:22.827038 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:22.828947 (+ 1909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:22.830926 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:22.830929 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:22.830932 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:22.831252 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:22.831755 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:22.831758 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:22.831760 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:22.831836 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:22.832384 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:22.832388 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:22.832478 (+ 90us) client_negotiation.cc:769] Sending connection context
0927 02:42:22.832641 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 02:42:22.833041 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":253,"spinlock_wait_cycles":16896,"thread_start_us":70,"threads_started":1}
I20260927 02:42:22.833220 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:22.820458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35073 (local address 127.0.98.2:35573)
0927 02:42:22.820841 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:22.820851 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:22.820887 (+ 36us) server_negotiation.cc:407] Connection header received
0927 02:42:22.821624 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:22.821631 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:22.821976 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:22.822218 (+ 242us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:22.823571 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:22.824496 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:22.825637 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:22.826086 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:22.826234 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:22.829119 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:22.829124 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:22.829136 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:22.829162 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:22.830825 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:22.831546 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:22.831552 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:22.831557 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:22.831671 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:22.831911 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:22.831914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:22.831916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:22.832286 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:22.832395 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:22.832760 (+ 365us) server_negotiation.cc:299] Negotiation successful
0927 02:42:22.833015 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":89,"threads_started":1}
W20260927 02:42:22.836486 16977 leader_election.cc:341] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e9a7c9d995ee4f51b1f1c8fe9292ba57 (127.0.98.2:35573): Not found: Tablet not found: 18ee83d58d224b038f82da4eaac1b961
W20260927 02:42:22.836653 16975 leader_election.cc:341] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 46370315edec4941bcd60a3c10b2b4a7 (127.0.98.3:37805): Not found: Tablet not found: 18ee83d58d224b038f82da4eaac1b961
I20260927 02:42:22.836755 16975 leader_election.cc:302] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fca1d04e37bd41b4a5792c073f2808ed; no voters: 46370315edec4941bcd60a3c10b2b4a7, e9a7c9d995ee4f51b1f1c8fe9292ba57
I20260927 02:42:22.836982 17362 raft_consensus.cc:2726] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 02:42:23.044111 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:23.043797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.1:33471)
0927 02:42:23.043954 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:23.044026 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 02:42:23.182878 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:23.182447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.2:42841)
0927 02:42:23.182717 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:23.182775 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":90,"threads_started":1}
I20260927 02:42:23.362542 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:23.362106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.3:38035)
0927 02:42:23.362401 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:23.362454 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":81,"threads_started":1}
I20260927 02:42:24.045221 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:24.044819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.1:45567)
0927 02:42:24.045049 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:24.045107 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":69,"threads_started":1}
I20260927 02:42:24.183984 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:24.183576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.2:53207)
0927 02:42:24.183833 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:24.183893 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":75,"threads_started":1}
I20260927 02:42:24.363644 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:24.363199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.3:42585)
0927 02:42:24.363477 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:24.363537 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":83,"threads_started":1}
I20260927 02:42:24.911725 17372 raft_consensus.cc:482] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:42:24.911847 17372 raft_consensus.cc:504] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46370315edec4941bcd60a3c10b2b4a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37805 } } peers { permanent_uuid: "e9a7c9d995ee4f51b1f1c8fe9292ba57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35573 } } peers { permanent_uuid: "fca1d04e37bd41b4a5792c073f2808ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39877 } }
I20260927 02:42:24.912204 17372 leader_election.cc:288] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46370315edec4941bcd60a3c10b2b4a7 (127.0.98.3:37805), e9a7c9d995ee4f51b1f1c8fe9292ba57 (127.0.98.2:35573)
W20260927 02:42:24.913008 16975 leader_election.cc:341] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 46370315edec4941bcd60a3c10b2b4a7 (127.0.98.3:37805): Not found: Tablet not found: 18ee83d58d224b038f82da4eaac1b961
W20260927 02:42:24.913122 16977 leader_election.cc:341] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e9a7c9d995ee4f51b1f1c8fe9292ba57 (127.0.98.2:35573): Not found: Tablet not found: 18ee83d58d224b038f82da4eaac1b961
I20260927 02:42:24.913180 16977 leader_election.cc:302] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fca1d04e37bd41b4a5792c073f2808ed; no voters: 46370315edec4941bcd60a3c10b2b4a7, e9a7c9d995ee4f51b1f1c8fe9292ba57
I20260927 02:42:24.913323 17372 raft_consensus.cc:2726] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 02:42:25.046345 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:25.045904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.1:52179)
0927 02:42:25.046151 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:25.046210 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":71,"threads_started":1}
I20260927 02:42:25.185109 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:25.184697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.2:58565)
0927 02:42:25.184967 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:25.185022 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":80,"threads_started":1}
I20260927 02:42:25.364843 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:25.364411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.3:53955)
0927 02:42:25.364681 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:25.364750 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":70,"threads_started":1}
I20260927 02:42:26.047603 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:26.047147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.1:55609)
0927 02:42:26.047441 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:26.047505 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":91,"threads_started":1}
I20260927 02:42:26.186316 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:26.185809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.2:39715)
0927 02:42:26.186125 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:26.186187 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":71,"threads_started":1}
I20260927 02:42:26.365908 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:26.365545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.3:37485)
0927 02:42:26.365768 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:26.365821 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":71,"threads_started":1}
I20260927 02:42:27.048702 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:27.048264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.1:34675)
0927 02:42:27.048542 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:27.048607 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":72,"threads_started":1}
I20260927 02:42:27.187366 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:27.186990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.2:46203)
0927 02:42:27.187223 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:27.187274 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":81,"threads_started":1}
I20260927 02:42:27.276643 17381 raft_consensus.cc:482] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:42:27.276775 17381 raft_consensus.cc:504] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46370315edec4941bcd60a3c10b2b4a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37805 } } peers { permanent_uuid: "e9a7c9d995ee4f51b1f1c8fe9292ba57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35573 } } peers { permanent_uuid: "fca1d04e37bd41b4a5792c073f2808ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39877 } }
I20260927 02:42:27.277197 17381 leader_election.cc:288] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46370315edec4941bcd60a3c10b2b4a7 (127.0.98.3:37805), e9a7c9d995ee4f51b1f1c8fe9292ba57 (127.0.98.2:35573)
W20260927 02:42:27.277987 16977 leader_election.cc:341] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e9a7c9d995ee4f51b1f1c8fe9292ba57 (127.0.98.2:35573): Not found: Tablet not found: 18ee83d58d224b038f82da4eaac1b961
W20260927 02:42:27.278095 16975 leader_election.cc:341] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 46370315edec4941bcd60a3c10b2b4a7 (127.0.98.3:37805): Not found: Tablet not found: 18ee83d58d224b038f82da4eaac1b961
I20260927 02:42:27.278164 16975 leader_election.cc:302] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fca1d04e37bd41b4a5792c073f2808ed; no voters: 46370315edec4941bcd60a3c10b2b4a7, e9a7c9d995ee4f51b1f1c8fe9292ba57
I20260927 02:42:27.278316 17381 raft_consensus.cc:2726] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 02:42:27.367089 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:27.366634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.3:46791)
0927 02:42:27.366923 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:27.366988 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":83,"threads_started":1}
I20260927 02:42:28.049816 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:28.049413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.1:53047)
0927 02:42:28.049661 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:28.049722 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":70,"threads_started":1}
I20260927 02:42:28.188580 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:28.188101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.2:51367)
0927 02:42:28.188408 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:28.188467 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":83,"threads_started":1}
I20260927 02:42:28.368283 17385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:28.367787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.3:41245)
0927 02:42:28.368138 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:28.368192 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":81,"threads_started":1}
I20260927 02:42:29.051007 17386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:29.050599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.1:56329)
0927 02:42:29.050847 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:29.050904 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":71,"threads_started":1}
I20260927 02:42:29.189810 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:29.189349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.2:44757)
0927 02:42:29.189634 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:29.189694 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":72,"threads_started":1}
I20260927 02:42:29.369405 17388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:29.368980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.3:37817)
0927 02:42:29.369255 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:29.369312 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":77,"threads_started":1}
I20260927 02:42:30.052119 17389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:30.051711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.1:55513)
0927 02:42:30.051964 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:30.052024 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":75,"threads_started":1}
I20260927 02:42:30.191021 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:30.190537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.2:55291)
0927 02:42:30.190804 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:30.190914 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":80,"threads_started":1}
I20260927 02:42:30.370550 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:30.370098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.3:37107)
0927 02:42:30.370402 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:30.370458 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":80,"threads_started":1}
I20260927 02:42:31.053313 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:31.052896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.1:46315)
0927 02:42:31.053163 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:31.053220 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":71,"threads_started":1}
I20260927 02:42:31.192226 17393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:31.191777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.2:55345)
0927 02:42:31.192069 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:31.192129 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":70,"threads_started":1}
I20260927 02:42:31.371696 17394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:31.371292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.3:50833)
0927 02:42:31.371547 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:31.371603 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":88,"threads_started":1}
I20260927 02:42:31.633285 17395 raft_consensus.cc:482] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:42:31.633431 17395 raft_consensus.cc:504] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46370315edec4941bcd60a3c10b2b4a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37805 } } peers { permanent_uuid: "e9a7c9d995ee4f51b1f1c8fe9292ba57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35573 } } peers { permanent_uuid: "fca1d04e37bd41b4a5792c073f2808ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39877 } }
I20260927 02:42:31.633843 17395 leader_election.cc:288] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46370315edec4941bcd60a3c10b2b4a7 (127.0.98.3:37805), e9a7c9d995ee4f51b1f1c8fe9292ba57 (127.0.98.2:35573)
W20260927 02:42:31.634717 16975 leader_election.cc:341] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 46370315edec4941bcd60a3c10b2b4a7 (127.0.98.3:37805): Not found: Tablet not found: 18ee83d58d224b038f82da4eaac1b961
W20260927 02:42:31.634912 16977 leader_election.cc:341] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e9a7c9d995ee4f51b1f1c8fe9292ba57 (127.0.98.2:35573): Not found: Tablet not found: 18ee83d58d224b038f82da4eaac1b961
I20260927 02:42:31.634997 16977 leader_election.cc:302] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fca1d04e37bd41b4a5792c073f2808ed; no voters: 46370315edec4941bcd60a3c10b2b4a7, e9a7c9d995ee4f51b1f1c8fe9292ba57
I20260927 02:42:31.635133 17395 raft_consensus.cc:2726] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 02:42:32.054486 17396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:32.054043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.1:55853)
0927 02:42:32.054321 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:32.054385 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":69,"threads_started":1}
I20260927 02:42:32.193361 17397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:32.192948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.2:34039)
0927 02:42:32.193207 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:32.193264 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":71,"threads_started":1}
I20260927 02:42:32.372983 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:32.372424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.3:54811)
0927 02:42:32.372753 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:32.372823 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":83,"threads_started":1}
I20260927 02:42:33.055625 17399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:33.055216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.1:54995)
0927 02:42:33.055477 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:33.055532 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":107,"threads_started":1}
I20260927 02:42:33.194556 17400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:33.194140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.2:57603)
0927 02:42:33.194391 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:33.194457 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":73,"threads_started":1}
I20260927 02:42:33.374056 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:33.373623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.3:36515)
0927 02:42:33.373897 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:33.373958 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":72,"threads_started":1}
I20260927 02:42:34.056726 17402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:34.056299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.1:59843)
0927 02:42:34.056562 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:34.056620 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":74,"threads_started":1}
I20260927 02:42:34.195643 17403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:34.195246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.2:57073)
0927 02:42:34.195495 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:34.195551 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":72,"threads_started":1}
I20260927 02:42:34.375300 17404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:34.374869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.3:38233)
0927 02:42:34.375129 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:34.375182 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":85,"threads_started":1}
I20260927 02:42:34.711063 17405 raft_consensus.cc:482] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:42:34.711199 17405 raft_consensus.cc:504] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46370315edec4941bcd60a3c10b2b4a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37805 } } peers { permanent_uuid: "e9a7c9d995ee4f51b1f1c8fe9292ba57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35573 } } peers { permanent_uuid: "fca1d04e37bd41b4a5792c073f2808ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39877 } }
I20260927 02:42:34.711602 17405 leader_election.cc:288] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46370315edec4941bcd60a3c10b2b4a7 (127.0.98.3:37805), e9a7c9d995ee4f51b1f1c8fe9292ba57 (127.0.98.2:35573)
W20260927 02:42:34.712352 16975 leader_election.cc:341] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 46370315edec4941bcd60a3c10b2b4a7 (127.0.98.3:37805): Not found: Tablet not found: 18ee83d58d224b038f82da4eaac1b961
W20260927 02:42:34.712493 16977 leader_election.cc:341] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e9a7c9d995ee4f51b1f1c8fe9292ba57 (127.0.98.2:35573): Not found: Tablet not found: 18ee83d58d224b038f82da4eaac1b961
I20260927 02:42:34.712584 16977 leader_election.cc:302] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fca1d04e37bd41b4a5792c073f2808ed; no voters: 46370315edec4941bcd60a3c10b2b4a7, e9a7c9d995ee4f51b1f1c8fe9292ba57
I20260927 02:42:34.712723 17405 raft_consensus.cc:2726] T 18ee83d58d224b038f82da4eaac1b961 P fca1d04e37bd41b4a5792c073f2808ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 02:42:35.057928 17406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:35.057418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.1:56155)
0927 02:42:35.057707 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:35.057825 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":88,"threads_started":1}
I20260927 02:42:35.196826 17407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:35.196397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.2:46269)
0927 02:42:35.196668 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:35.196731 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":71,"threads_started":1}
I20260927 02:42:35.376362 17408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:35.375983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.3:54153)
0927 02:42:35.376213 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:35.376269 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":75,"threads_started":1}
I20260927 02:42:36.059288 17409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:36.058868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.1:44329)
0927 02:42:36.059135 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:36.059193 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":73,"threads_started":1}
I20260927 02:42:36.198194 17410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:36.197718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42269 (local address 127.0.98.2:50201)
0927 02:42:36.198041 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:36.198098 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":136,"threads_started":1}
I20260927 02:42:36.359658 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 16919
I20260927 02:42:36.364845 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 17092
I20260927 02:42:36.369539 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 17226
2026-09-27T02:42:36Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59153 ms)
[----------] 30 tests from SecurityITest (238339 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:42:36 dist-test-slave-zv8z krb5kdc[17413](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:42:36 dist-test-slave-zv8z krb5kdc[17413](info): set up 2 sockets
Sep 27 02:42:36 dist-test-slave-zv8z krb5kdc[17413](info): commencing operation
krb5kdc: starting...
W20260927 02:42:37.337949 392 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/krb5kdc/test-user.keytab.
Sep 27 02:42:37 dist-test-slave-zv8z krb5kdc[17413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:42:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:42:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:42:37.451112 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43259
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43259
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:37.557616 17426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:37.557891 17426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:37.557930 17426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:37.561954 17426 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:42:37.562028 17426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:37.562050 17426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:37.562067 17426 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:42:37.562083 17426 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:42:37.567472 17426 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:42:37.568477 17426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:37.569479 17426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:37.578105 17434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:37.578066 17432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:37.578132 17431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:37.578117 17426 server_base.cc:1034] running on GCE node
I20260927 02:42:37.578953 17426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:37.580054 17426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:37.581221 17426 hybrid_clock.cc:648] HybridClock initialized: now 1790476957581194 us; error 37 us; skew 500 ppm
Sep 27 02:42:37 dist-test-slave-zv8z krb5kdc[17413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476957, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:37.584007 17426 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:42:37.584858 17426 webserver.cc:460] Webserver started at http://127.0.98.62:44497/ using document root <none> and password file <none>
I20260927 02:42:37.585356 17426 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:37.585412 17426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:37.585566 17426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:37.587611 17426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/master-0/data/instance:
uuid: "5c610f4916aa403eb1a55ce83f9517e5"
format_stamp: "Formatted at 2026-09-27 02:42:37 on dist-test-slave-zv8z"
server_key: "4ba90f6845d7856d30e7553321c9657a"
server_key_iv: "97d3bf1e3eab0b168bf1d1ebbc6adf9f"
server_key_version: "encryptionkey@0"
I20260927 02:42:37.588097 17426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/master-0/wal/instance:
uuid: "5c610f4916aa403eb1a55ce83f9517e5"
format_stamp: "Formatted at 2026-09-27 02:42:37 on dist-test-slave-zv8z"
server_key: "4ba90f6845d7856d30e7553321c9657a"
server_key_iv: "97d3bf1e3eab0b168bf1d1ebbc6adf9f"
server_key_version: "encryptionkey@0"
I20260927 02:42:37.591570 17426 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:42:37.593886 17441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:37.595124 17426 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:42:37.595228 17426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "5c610f4916aa403eb1a55ce83f9517e5"
format_stamp: "Formatted at 2026-09-27 02:42:37 on dist-test-slave-zv8z"
server_key: "4ba90f6845d7856d30e7553321c9657a"
server_key_iv: "97d3bf1e3eab0b168bf1d1ebbc6adf9f"
server_key_version: "encryptionkey@0"
I20260927 02:42:37.595329 17426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:37.606359 17426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:37.608976 17426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:37.609141 17426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:37.615854 17426 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43259
I20260927 02:42:37.615871 17493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43259 every 8 connection(s)
I20260927 02:42:37.616775 17426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:42:37.619418 17494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:37.624639 17494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c610f4916aa403eb1a55ce83f9517e5: Bootstrap starting.
I20260927 02:42:37.626720 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 17426
I20260927 02:42:37.626878 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/master-0/wal/instance
I20260927 02:42:37.627020 17494 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c610f4916aa403eb1a55ce83f9517e5: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:37.627182 392 external_mini_cluster.cc:1428] Setting key 618325426ffdaf471acd7f190be34f50
I20260927 02:42:37.627821 17494 log.cc:824] T 00000000000000000000000000000000 P 5c610f4916aa403eb1a55ce83f9517e5: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:37.630007 17494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c610f4916aa403eb1a55ce83f9517e5: No bootstrap required, opened a new log
I20260927 02:42:37.634104 17494 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c610f4916aa403eb1a55ce83f9517e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c610f4916aa403eb1a55ce83f9517e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43259 } }
I20260927 02:42:37.634404 17494 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c610f4916aa403eb1a55ce83f9517e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:37.634465 17494 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c610f4916aa403eb1a55ce83f9517e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c610f4916aa403eb1a55ce83f9517e5, State: Initialized, Role: FOLLOWER
I20260927 02:42:37.634877 17494 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c610f4916aa403eb1a55ce83f9517e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c610f4916aa403eb1a55ce83f9517e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43259 } }
I20260927 02:42:37.635001 17494 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c610f4916aa403eb1a55ce83f9517e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:42:37.635062 17494 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c610f4916aa403eb1a55ce83f9517e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:42:37.635150 17494 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c610f4916aa403eb1a55ce83f9517e5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:37.635980 17494 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c610f4916aa403eb1a55ce83f9517e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c610f4916aa403eb1a55ce83f9517e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43259 } }
Sep 27 02:42:37 dist-test-slave-zv8z krb5kdc[17413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:42:37.636278 17494 leader_election.cc:302] T 00000000000000000000000000000000 P 5c610f4916aa403eb1a55ce83f9517e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c610f4916aa403eb1a55ce83f9517e5; no voters:
I20260927 02:42:37.636548 17494 leader_election.cc:288] T 00000000000000000000000000000000 P 5c610f4916aa403eb1a55ce83f9517e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:42:37.636623 17499 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c610f4916aa403eb1a55ce83f9517e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:37.636794 17499 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c610f4916aa403eb1a55ce83f9517e5 [term 1 LEADER]: Becoming Leader. State: Replica: 5c610f4916aa403eb1a55ce83f9517e5, State: Running, Role: LEADER
I20260927 02:42:37.637166 17499 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c610f4916aa403eb1a55ce83f9517e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c610f4916aa403eb1a55ce83f9517e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43259 } }
I20260927 02:42:37.637797 17494 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c610f4916aa403eb1a55ce83f9517e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:42:37.638355 17499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c610f4916aa403eb1a55ce83f9517e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c610f4916aa403eb1a55ce83f9517e5. Latest consensus state: current_term: 1 leader_uuid: "5c610f4916aa403eb1a55ce83f9517e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c610f4916aa403eb1a55ce83f9517e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43259 } } }
I20260927 02:42:37.638463 17499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c610f4916aa403eb1a55ce83f9517e5 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:37.640168 17503 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:42:37.640856 17500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c610f4916aa403eb1a55ce83f9517e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c610f4916aa403eb1a55ce83f9517e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c610f4916aa403eb1a55ce83f9517e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43259 } } }
I20260927 02:42:37.640962 17500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c610f4916aa403eb1a55ce83f9517e5 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:37.641785 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:37.628456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42638 (local address 127.0.98.62:43259)
0927 02:42:37.628843 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:37.628861 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:37.628912 (+ 51us) server_negotiation.cc:407] Connection header received
0927 02:42:37.629701 (+ 789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:37.629724 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:37.630120 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:37.630496 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:37.631112 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.632572 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.633413 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.634097 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.634275 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:37.636930 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:37.636936 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:37.636952 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:37.636982 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:37.638850 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:37.639351 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:37.639356 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:37.639362 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:37.639496 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:37.639796 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:37.639800 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:37.639802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:37.640304 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:37.640494 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:37.640752 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 02:42:37.641040 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":105,"threads_started":1}
I20260927 02:42:37.643325 17503 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:42:37.648645 17503 catalog_manager.cc:1347] Generated new cluster ID: 9decbaa2ac514a7998283362e63262f6
I20260927 02:42:37.648695 17503 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:42:37.666903 17503 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:42:37.668484 17503 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:42:37.685390 17503 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c610f4916aa403eb1a55ce83f9517e5: Generated new TSK 0
I20260927 02:42:37.686062 17503 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:42:37.716969 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43259
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:37.835862 17520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:37.836086 17520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:37.836124 17520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:37.840070 17520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:37.840121 17520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:37.840188 17520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:42:37.845203 17520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:42:37.846119 17520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:37.847087 17520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:37.856117 17526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:37.856266 17525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:37.856113 17528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:37.856387 17520 server_base.cc:1034] running on GCE node
I20260927 02:42:37.856875 17520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:37.857470 17520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:37.858616 17520 hybrid_clock.cc:648] HybridClock initialized: now 1790476957858596 us; error 30 us; skew 500 ppm
Sep 27 02:42:37 dist-test-slave-zv8z krb5kdc[17413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476957, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:37.861325 17520 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:42:37.862100 17520 webserver.cc:460] Webserver started at http://127.0.98.1:33283/ using document root <none> and password file <none>
I20260927 02:42:37.862632 17520 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:37.862694 17520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:37.862906 17520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:37.864797 17520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/ts-0/data/instance:
uuid: "047b1cf367ea48838147509edb136791"
format_stamp: "Formatted at 2026-09-27 02:42:37 on dist-test-slave-zv8z"
server_key: "f073266269342b7a1096c5679427c4a9"
server_key_iv: "90d04c7bdfbf2ce0aa3ba431df523c56"
server_key_version: "encryptionkey@0"
I20260927 02:42:37.865305 17520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/ts-0/wal/instance:
uuid: "047b1cf367ea48838147509edb136791"
format_stamp: "Formatted at 2026-09-27 02:42:37 on dist-test-slave-zv8z"
server_key: "f073266269342b7a1096c5679427c4a9"
server_key_iv: "90d04c7bdfbf2ce0aa3ba431df523c56"
server_key_version: "encryptionkey@0"
I20260927 02:42:37.868714 17520 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 02:42:37.871002 17535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:37.872099 17520 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:42:37.872196 17520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/ts-0/wal
uuid: "047b1cf367ea48838147509edb136791"
format_stamp: "Formatted at 2026-09-27 02:42:37 on dist-test-slave-zv8z"
server_key: "f073266269342b7a1096c5679427c4a9"
server_key_iv: "90d04c7bdfbf2ce0aa3ba431df523c56"
server_key_version: "encryptionkey@0"
I20260927 02:42:37.872282 17520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:37.893330 17520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:37.896107 17520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:37.896268 17520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:37.896798 17520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:37.897543 17520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:37.897588 17520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:37.897639 17520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:37.897665 17520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:37.905902 17520 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40991
I20260927 02:42:37.905978 17648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40991 every 8 connection(s)
I20260927 02:42:37.906906 17520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:42:37.913095 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 17520
I20260927 02:42:37.913233 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:42:37.913533 392 external_mini_cluster.cc:1428] Setting key da590c48431e01503abcef4dbe0dee83
Sep 27 02:42:37 dist-test-slave-zv8z krb5kdc[17413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476957, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:42:37.920018 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:37.908632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41793 (local address 127.0.98.62:43259)
0927 02:42:37.908816 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:37.908820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:37.909581 (+ 761us) server_negotiation.cc:407] Connection header received
0927 02:42:37.910807 (+ 1226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:37.910810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:37.910863 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:37.910936 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:37.912032 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.912562 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.914037 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.914314 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.914404 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:37.916621 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:37.916625 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:37.916628 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:37.916655 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:37.918082 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:37.918632 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:37.918636 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:37.918637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:37.918718 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:37.919015 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:37.919019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:37.919021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:37.919204 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:37.919287 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:37.919743 (+ 456us) server_negotiation.cc:299] Negotiation successful
0927 02:42:37.919859 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 02:42:37.920748 17651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:37.908969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43259 (local address 127.0.98.1:41793)
0927 02:42:37.909426 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:37.909466 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:37.910596 (+ 1130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:37.911103 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:37.911110 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:37.911434 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:37.911892 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:37.911904 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.912677 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.912680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:37.913930 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:37.913939 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:37.914442 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:37.914458 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:37.914613 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:37.915133 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:37.915149 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:37.916467 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:37.918193 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:37.918199 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:37.918212 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:37.918542 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:37.918797 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:37.918800 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:37.918802 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:37.918932 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:37.919300 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:37.919306 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:37.919524 (+ 218us) client_negotiation.cc:769] Sending connection context
0927 02:42:37.919677 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 02:42:37.919972 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":104,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:42:37.921995 17649 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43259
I20260927 02:42:37.922204 17649 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:37.922644 17649 heartbeater.cc:511] Master 127.0.98.62:43259 requested a full tablet report, sending...
I20260927 02:42:37.924327 17458 ts_manager.cc:194] Registered new tserver with Master: 047b1cf367ea48838147509edb136791 (127.0.98.1:40991)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 02:42:37.925462 17458 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41793
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:42:37.937927 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43259
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:38.042456 17654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:38.042707 17654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:38.042768 17654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:38.046663 17654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:38.046725 17654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:38.046801 17654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:42:38.051797 17654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:42:38.052809 17654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:38.053767 17654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:38.062681 17662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:38.062798 17659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:38.062901 17654 server_base.cc:1034] running on GCE node
W20260927 02:42:38.062706 17660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:38.063344 17654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:38.063912 17654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:38.065088 17654 hybrid_clock.cc:648] HybridClock initialized: now 1790476958065059 us; error 40 us; skew 500 ppm
Sep 27 02:42:38 dist-test-slave-zv8z krb5kdc[17413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476958, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:38.067701 17654 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:42:38.068512 17654 webserver.cc:460] Webserver started at http://127.0.98.2:40501/ using document root <none> and password file <none>
I20260927 02:42:38.069025 17654 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:38.069089 17654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:38.069298 17654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:38.071275 17654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/ts-1/data/instance:
uuid: "f40daf5e697343ca91f724ea16a61fb5"
format_stamp: "Formatted at 2026-09-27 02:42:38 on dist-test-slave-zv8z"
server_key: "b8274542159f0ac9b01f654c8e7182d1"
server_key_iv: "86703fed9072a5a2ef9f5e1faf5e8667"
server_key_version: "encryptionkey@0"
I20260927 02:42:38.071782 17654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f40daf5e697343ca91f724ea16a61fb5"
format_stamp: "Formatted at 2026-09-27 02:42:38 on dist-test-slave-zv8z"
server_key: "b8274542159f0ac9b01f654c8e7182d1"
server_key_iv: "86703fed9072a5a2ef9f5e1faf5e8667"
server_key_version: "encryptionkey@0"
I20260927 02:42:38.075278 17654 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 02:42:38.077538 17669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:38.078707 17654 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:42:38.078814 17654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/ts-1/wal
uuid: "f40daf5e697343ca91f724ea16a61fb5"
format_stamp: "Formatted at 2026-09-27 02:42:38 on dist-test-slave-zv8z"
server_key: "b8274542159f0ac9b01f654c8e7182d1"
server_key_iv: "86703fed9072a5a2ef9f5e1faf5e8667"
server_key_version: "encryptionkey@0"
I20260927 02:42:38.078907 17654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:38.094713 17654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:38.097524 17654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:38.097695 17654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:38.098336 17654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:38.099169 17654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:38.099223 17654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:38.099284 17654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:38.099313 17654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:38.107981 17654 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37559
I20260927 02:42:38.108013 17782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37559 every 8 connection(s)
I20260927 02:42:38.108851 17654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:42:38.113551 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 17654
I20260927 02:42:38.113682 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:42:38.113984 392 external_mini_cluster.cc:1428] Setting key 920d6f683fb520e39a354f66a45ba8fb
Sep 27 02:42:38 dist-test-slave-zv8z krb5kdc[17413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476958, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:42:38.123121 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:38.110775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52387 (local address 127.0.98.62:43259)
0927 02:42:38.110911 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:38.110915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:38.111808 (+ 893us) server_negotiation.cc:407] Connection header received
0927 02:42:38.113006 (+ 1198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:38.113008 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:38.113053 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:38.113125 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:38.114731 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:38.115322 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:38.116480 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:38.116790 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:38.116894 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:38.119525 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:38.119529 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:38.119532 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:38.119561 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:38.121106 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:38.121595 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:38.121599 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:38.121601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:38.121694 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:38.121988 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:38.121991 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:38.121993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:38.122213 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:38.122361 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:38.122819 (+ 458us) server_negotiation.cc:299] Negotiation successful
0927 02:42:38.122954 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:42:38.123826 17785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:38.111215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43259 (local address 127.0.98.2:52387)
0927 02:42:38.111658 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:38.111701 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:38.112810 (+ 1109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:38.113402 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:38.113411 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:38.113927 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:38.114584 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:38.114603 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:38.115439 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:38.115443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:38.116357 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:38.116366 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:38.116943 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:38.116947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:38.117143 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:38.117762 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:38.117782 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:38.119365 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:38.121225 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:38.121230 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:38.121241 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:38.121503 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:38.121775 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:38.121776 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:38.121778 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:38.121905 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:38.122361 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:38.122366 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:38.122589 (+ 223us) client_negotiation.cc:769] Sending connection context
0927 02:42:38.122739 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 02:42:38.122978 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":104,"threads_started":1}
I20260927 02:42:38.125046 17783 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43259
I20260927 02:42:38.125255 17783 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:38.125785 17783 heartbeater.cc:511] Master 127.0.98.62:43259 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 02:42:38.126811 17458 ts_manager.cc:194] Registered new tserver with Master: f40daf5e697343ca91f724ea16a61fb5 (127.0.98.2:37559)
I20260927 02:42:38.127363 17458 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52387
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:42:38.140712 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43259
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:38.247166 17788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:38.247411 17788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:38.247449 17788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:38.251375 17788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:38.251426 17788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:38.251482 17788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:42:38.256716 17788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:42:38.257659 17788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:38.258647 17788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:38.267795 17793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:38.267828 17794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:38.267901 17788 server_base.cc:1034] running on GCE node
W20260927 02:42:38.267830 17796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:38.268390 17788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:38.269001 17788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:38.270171 17788 hybrid_clock.cc:648] HybridClock initialized: now 1790476958270135 us; error 44 us; skew 500 ppm
Sep 27 02:42:38 dist-test-slave-zv8z krb5kdc[17413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476958, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:38.272804 17788 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:42:38.273630 17788 webserver.cc:460] Webserver started at http://127.0.98.3:43435/ using document root <none> and password file <none>
I20260927 02:42:38.274140 17788 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:38.274232 17788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:38.274461 17788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:38.276515 17788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/ts-2/data/instance:
uuid: "73535a3974814af1a054df0689153faf"
format_stamp: "Formatted at 2026-09-27 02:42:38 on dist-test-slave-zv8z"
server_key: "b0b4d53cca90cb7dfdd8d5f051740b7c"
server_key_iv: "2039ee296a7f0183283c3a80be312130"
server_key_version: "encryptionkey@0"
I20260927 02:42:38.277040 17788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/ts-2/wal/instance:
uuid: "73535a3974814af1a054df0689153faf"
format_stamp: "Formatted at 2026-09-27 02:42:38 on dist-test-slave-zv8z"
server_key: "b0b4d53cca90cb7dfdd8d5f051740b7c"
server_key_iv: "2039ee296a7f0183283c3a80be312130"
server_key_version: "encryptionkey@0"
I20260927 02:42:38.280521 17788 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 02:42:38.282974 17803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:38.284217 17788 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:42:38.284325 17788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/ts-2/wal
uuid: "73535a3974814af1a054df0689153faf"
format_stamp: "Formatted at 2026-09-27 02:42:38 on dist-test-slave-zv8z"
server_key: "b0b4d53cca90cb7dfdd8d5f051740b7c"
server_key_iv: "2039ee296a7f0183283c3a80be312130"
server_key_version: "encryptionkey@0"
I20260927 02:42:38.284416 17788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:38.302227 17788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:38.305359 17788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:38.305542 17788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:38.306119 17788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:38.306988 17788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:38.307039 17788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:38.307090 17788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:38.307119 17788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:38.315457 17788 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42589
I20260927 02:42:38.315490 17916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42589 every 8 connection(s)
I20260927 02:42:38.316283 17788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:42:38.316704 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 17788
I20260927 02:42:38.316807 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:42:38.317071 392 external_mini_cluster.cc:1428] Setting key 9a9eff16e0bae157d7f2ffda7b5e2156
Sep 27 02:42:38 dist-test-slave-zv8z krb5kdc[17413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476958, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:42:38.328989 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:38.318100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37329 (local address 127.0.98.62:43259)
0927 02:42:38.318207 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:38.318212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:38.319197 (+ 985us) server_negotiation.cc:407] Connection header received
0927 02:42:38.320472 (+ 1275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:38.320475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:38.320528 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:38.320596 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:38.321711 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:38.322232 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:38.323335 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:38.323556 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:38.323641 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:38.325895 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:38.325899 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:38.325901 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:38.325929 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:38.327171 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:38.327613 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:38.327616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:38.327618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:38.327691 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:38.327989 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:38.327992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:38.327994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:38.328173 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:38.328254 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:38.328715 (+ 461us) server_negotiation.cc:299] Negotiation successful
0927 02:42:38.328829 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 02:42:38.329731 17919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:38.318622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43259 (local address 127.0.98.3:37329)
0927 02:42:38.319066 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:38.319106 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:38.320276 (+ 1170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:38.320763 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:38.320770 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:38.321110 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:38.321581 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:38.321594 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:38.322356 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:38.322360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:38.323225 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:38.323235 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:38.323683 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:38.323687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:38.323856 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:38.324356 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:38.324405 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:38.325741 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:38.327273 (+ 1532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:38.327278 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:38.327292 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:38.327523 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:38.327769 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:38.327772 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:38.327774 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:38.327904 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:38.328268 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:38.328274 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:38.328499 (+ 225us) client_negotiation.cc:769] Sending connection context
0927 02:42:38.328656 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 02:42:38.328965 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":121,"threads_started":1}
I20260927 02:42:38.330793 17917 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43259
I20260927 02:42:38.331013 17917 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:38.331518 17917 heartbeater.cc:511] Master 127.0.98.62:43259 requested a full tablet report, sending...
I20260927 02:42:38.332485 17457 ts_manager.cc:194] Registered new tserver with Master: 73535a3974814af1a054df0689153faf (127.0.98.3:42589)
I20260927 02:42:38.333104 17457 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37329
I20260927 02:42:38.343327 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:42:38.352505 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:38.345271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42640 (local address 127.0.98.62:43259)
0927 02:42:38.345429 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:38.345433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:38.345539 (+ 106us) server_negotiation.cc:407] Connection header received
0927 02:42:38.345647 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:38.345649 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:38.345689 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:38.345744 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:38.346378 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:38.346945 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:38.347817 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:38.348031 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:38.348119 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:38.349463 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:38.349466 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:38.349468 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:38.349492 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:38.350805 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:38.351267 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:38.351270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:38.351271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:38.351357 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:38.351624 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:38.351627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:38.351628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:38.351904 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:38.352027 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:38.352274 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 02:42:38.352375 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 02:42:38.355600 17457 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42640:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:42:38.358675 17457 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:42:38.373087 17929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:38.367008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40991 (local address 127.0.0.1:56704)
0927 02:42:38.367389 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:38.367440 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:38.367561 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:38.368500 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:38.368506 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:38.368530 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:38.368660 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:38.368667 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:38.370481 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:38.370485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:38.371526 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:38.371535 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:38.372562 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:38.372566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:38.372677 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:38.372719 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 02:42:38.372814 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 02:42:38.372931 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":58,"thread_start_us":69,"threads_started":1}
I20260927 02:42:38.373116 17930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:38.367173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56704 (local address 127.0.98.1:40991)
0927 02:42:38.367993 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:38.368011 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:38.368035 (+ 24us) server_negotiation.cc:407] Connection header received
0927 02:42:38.368113 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:38.368118 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:38.368329 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:38.368818 (+ 489us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:38.368962 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:38.370343 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:38.371629 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:38.372444 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:38.372551 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:38.372693 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:38.372864 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 02:42:38.372959 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":714,"thread_start_us":118,"threads_started":1}
I20260927 02:42:38.373693 17928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:38.367024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37559 (local address 127.0.0.1:40654)
0927 02:42:38.368002 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:38.368015 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:38.368096 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:38.368696 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:38.368699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:38.368715 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:38.368801 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:38.368807 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:38.370209 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:38.370213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:38.371601 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:38.371611 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:38.373161 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:38.373164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:38.373256 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:38.373288 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 02:42:38.373349 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 02:42:38.373409 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":921,"mutex_wait_us":4,"thread_start_us":106,"threads_started":1}
I20260927 02:42:38.373888 17932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:38.367804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40654 (local address 127.0.98.2:37559)
0927 02:42:38.368334 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:38.368346 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:38.368368 (+ 22us) server_negotiation.cc:407] Connection header received
0927 02:42:38.368437 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:38.368441 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:38.368585 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:38.368703 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:38.368919 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:38.370090 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:38.371708 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:38.372452 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:38.372551 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:38.372693 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:38.373650 (+ 957us) server_negotiation.cc:299] Negotiation successful
0927 02:42:38.373748 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":98,"threads_started":1}
I20260927 02:42:38.373471 17931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:38.367716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42589 (local address 127.0.0.1:33178)
0927 02:42:38.368921 (+ 1205us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:38.368934 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:38.369034 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:38.369602 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:38.369605 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:38.369621 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:38.369700 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:38.369706 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:38.371020 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:38.371023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:38.372024 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:38.372033 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:38.372839 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:38.372842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:38.372938 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:38.372960 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 02:42:38.373037 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 02:42:38.373101 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1145,"mutex_wait_us":4,"thread_start_us":96,"threads_started":1}
I20260927 02:42:38.376075 17583 tablet_service.cc:1467] Processing CreateTablet for tablet 12dda33bdb59404cb6fcb4829ddd99ea (DEFAULT_TABLE table=test-table [id=aef045ec939f46448fc4e44947775802]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:38.376565 17717 tablet_service.cc:1467] Processing CreateTablet for tablet 12dda33bdb59404cb6fcb4829ddd99ea (DEFAULT_TABLE table=test-table [id=aef045ec939f46448fc4e44947775802]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:38.377015 17583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12dda33bdb59404cb6fcb4829ddd99ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:38.377493 17717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12dda33bdb59404cb6fcb4829ddd99ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:38.381759 17933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:38.368163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33178 (local address 127.0.98.3:42589)
0927 02:42:38.368532 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:38.368544 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:38.369024 (+ 480us) server_negotiation.cc:407] Connection header received
0927 02:42:38.369127 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:38.369132 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:38.369314 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:38.369449 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:38.369823 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:38.370912 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:38.372127 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:38.372713 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:38.372834 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:38.372961 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:38.373062 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 02:42:38.373147 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":162,"threads_started":1}
I20260927 02:42:38.383742 17935 tablet_bootstrap.cc:492] T 12dda33bdb59404cb6fcb4829ddd99ea P f40daf5e697343ca91f724ea16a61fb5: Bootstrap starting.
I20260927 02:42:38.383742 17934 tablet_bootstrap.cc:492] T 12dda33bdb59404cb6fcb4829ddd99ea P 047b1cf367ea48838147509edb136791: Bootstrap starting.
I20260927 02:42:38.384516 17851 tablet_service.cc:1467] Processing CreateTablet for tablet 12dda33bdb59404cb6fcb4829ddd99ea (DEFAULT_TABLE table=test-table [id=aef045ec939f46448fc4e44947775802]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:38.385365 17851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12dda33bdb59404cb6fcb4829ddd99ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:38.385958 17934 tablet_bootstrap.cc:654] T 12dda33bdb59404cb6fcb4829ddd99ea P 047b1cf367ea48838147509edb136791: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:38.385958 17935 tablet_bootstrap.cc:654] T 12dda33bdb59404cb6fcb4829ddd99ea P f40daf5e697343ca91f724ea16a61fb5: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:38.387168 17935 log.cc:824] T 12dda33bdb59404cb6fcb4829ddd99ea P f40daf5e697343ca91f724ea16a61fb5: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:38.387168 17934 log.cc:824] T 12dda33bdb59404cb6fcb4829ddd99ea P 047b1cf367ea48838147509edb136791: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:38.389243 17935 tablet_bootstrap.cc:492] T 12dda33bdb59404cb6fcb4829ddd99ea P f40daf5e697343ca91f724ea16a61fb5: No bootstrap required, opened a new log
I20260927 02:42:38.389439 17935 ts_tablet_manager.cc:1397] T 12dda33bdb59404cb6fcb4829ddd99ea P f40daf5e697343ca91f724ea16a61fb5: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260927 02:42:38.389492 17934 tablet_bootstrap.cc:492] T 12dda33bdb59404cb6fcb4829ddd99ea P 047b1cf367ea48838147509edb136791: No bootstrap required, opened a new log
I20260927 02:42:38.389644 17934 ts_tablet_manager.cc:1397] T 12dda33bdb59404cb6fcb4829ddd99ea P 047b1cf367ea48838147509edb136791: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 02:42:38.391290 17938 tablet_bootstrap.cc:492] T 12dda33bdb59404cb6fcb4829ddd99ea P 73535a3974814af1a054df0689153faf: Bootstrap starting.
I20260927 02:42:38.393076 17938 tablet_bootstrap.cc:654] T 12dda33bdb59404cb6fcb4829ddd99ea P 73535a3974814af1a054df0689153faf: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:38.393358 17935 raft_consensus.cc:348] T 12dda33bdb59404cb6fcb4829ddd99ea P f40daf5e697343ca91f724ea16a61fb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047b1cf367ea48838147509edb136791" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40991 } } peers { permanent_uuid: "f40daf5e697343ca91f724ea16a61fb5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37559 } } peers { permanent_uuid: "73535a3974814af1a054df0689153faf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42589 } }
I20260927 02:42:38.393358 17934 raft_consensus.cc:348] T 12dda33bdb59404cb6fcb4829ddd99ea P 047b1cf367ea48838147509edb136791 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047b1cf367ea48838147509edb136791" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40991 } } peers { permanent_uuid: "f40daf5e697343ca91f724ea16a61fb5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37559 } } peers { permanent_uuid: "73535a3974814af1a054df0689153faf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42589 } }
I20260927 02:42:38.393617 17934 raft_consensus.cc:374] T 12dda33bdb59404cb6fcb4829ddd99ea P 047b1cf367ea48838147509edb136791 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:38.393618 17935 raft_consensus.cc:374] T 12dda33bdb59404cb6fcb4829ddd99ea P f40daf5e697343ca91f724ea16a61fb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:38.393673 17935 raft_consensus.cc:729] T 12dda33bdb59404cb6fcb4829ddd99ea P f40daf5e697343ca91f724ea16a61fb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f40daf5e697343ca91f724ea16a61fb5, State: Initialized, Role: FOLLOWER
I20260927 02:42:38.393699 17934 raft_consensus.cc:729] T 12dda33bdb59404cb6fcb4829ddd99ea P 047b1cf367ea48838147509edb136791 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 047b1cf367ea48838147509edb136791, State: Initialized, Role: FOLLOWER
I20260927 02:42:38.394099 17938 log.cc:824] T 12dda33bdb59404cb6fcb4829ddd99ea P 73535a3974814af1a054df0689153faf: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:38.394150 17935 consensus_queue.cc:260] T 12dda33bdb59404cb6fcb4829ddd99ea P f40daf5e697343ca91f724ea16a61fb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047b1cf367ea48838147509edb136791" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40991 } } peers { permanent_uuid: "f40daf5e697343ca91f724ea16a61fb5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37559 } } peers { permanent_uuid: "73535a3974814af1a054df0689153faf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42589 } }
I20260927 02:42:38.394150 17934 consensus_queue.cc:260] T 12dda33bdb59404cb6fcb4829ddd99ea P 047b1cf367ea48838147509edb136791 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047b1cf367ea48838147509edb136791" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40991 } } peers { permanent_uuid: "f40daf5e697343ca91f724ea16a61fb5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37559 } } peers { permanent_uuid: "73535a3974814af1a054df0689153faf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42589 } }
I20260927 02:42:38.395143 17935 ts_tablet_manager.cc:1428] T 12dda33bdb59404cb6fcb4829ddd99ea P f40daf5e697343ca91f724ea16a61fb5: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 02:42:38.395145 17934 ts_tablet_manager.cc:1428] T 12dda33bdb59404cb6fcb4829ddd99ea P 047b1cf367ea48838147509edb136791: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 02:42:38.395470 17649 heartbeater.cc:503] Master 127.0.98.62:43259 was elected leader, sending a full tablet report...
I20260927 02:42:38.395470 17783 heartbeater.cc:503] Master 127.0.98.62:43259 was elected leader, sending a full tablet report...
I20260927 02:42:38.395998 17938 tablet_bootstrap.cc:492] T 12dda33bdb59404cb6fcb4829ddd99ea P 73535a3974814af1a054df0689153faf: No bootstrap required, opened a new log
I20260927 02:42:38.396147 17938 ts_tablet_manager.cc:1397] T 12dda33bdb59404cb6fcb4829ddd99ea P 73535a3974814af1a054df0689153faf: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 02:42:38.399113 17938 raft_consensus.cc:348] T 12dda33bdb59404cb6fcb4829ddd99ea P 73535a3974814af1a054df0689153faf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047b1cf367ea48838147509edb136791" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40991 } } peers { permanent_uuid: "f40daf5e697343ca91f724ea16a61fb5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37559 } } peers { permanent_uuid: "73535a3974814af1a054df0689153faf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42589 } }
I20260927 02:42:38.399375 17938 raft_consensus.cc:374] T 12dda33bdb59404cb6fcb4829ddd99ea P 73535a3974814af1a054df0689153faf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:38.399434 17938 raft_consensus.cc:729] T 12dda33bdb59404cb6fcb4829ddd99ea P 73535a3974814af1a054df0689153faf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73535a3974814af1a054df0689153faf, State: Initialized, Role: FOLLOWER
I20260927 02:42:38.399787 17938 consensus_queue.cc:260] T 12dda33bdb59404cb6fcb4829ddd99ea P 73535a3974814af1a054df0689153faf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047b1cf367ea48838147509edb136791" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40991 } } peers { permanent_uuid: "f40daf5e697343ca91f724ea16a61fb5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37559 } } peers { permanent_uuid: "73535a3974814af1a054df0689153faf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42589 } }
I20260927 02:42:38.400339 17917 heartbeater.cc:503] Master 127.0.98.62:43259 was elected leader, sending a full tablet report...
I20260927 02:42:38.400641 17938 ts_tablet_manager.cc:1428] T 12dda33bdb59404cb6fcb4829ddd99ea P 73535a3974814af1a054df0689153faf: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20260927 02:42:38.407919 17650 tablet.cc:2369] T 12dda33bdb59404cb6fcb4829ddd99ea P 047b1cf367ea48838147509edb136791: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:42:38.500372 17942 raft_consensus.cc:482] T 12dda33bdb59404cb6fcb4829ddd99ea P 73535a3974814af1a054df0689153faf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:42:38.500566 17942 raft_consensus.cc:504] T 12dda33bdb59404cb6fcb4829ddd99ea P 73535a3974814af1a054df0689153faf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047b1cf367ea48838147509edb136791" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40991 } } peers { permanent_uuid: "f40daf5e697343ca91f724ea16a61fb5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37559 } } peers { permanent_uuid: "73535a3974814af1a054df0689153faf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42589 } }
I20260927 02:42:38.501715 17942 leader_election.cc:288] T 12dda33bdb59404cb6fcb4829ddd99ea P 73535a3974814af1a054df0689153faf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 047b1cf367ea48838147509edb136791 (127.0.98.1:40991), f40daf5e697343ca91f724ea16a61fb5 (127.0.98.2:37559)
I20260927 02:42:38.506534 17919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:38.501994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37559 (local address 127.0.98.3:39967)
0927 02:42:38.502163 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:38.502176 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:38.502287 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:38.502542 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:38.502544 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:38.502557 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:38.502667 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:38.502673 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:38.503695 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:38.503699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:38.505057 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:38.505068 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:38.505999 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:38.506003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:38.506144 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:38.506181 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 02:42:38.506281 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 02:42:38.506382 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":42}
I20260927 02:42:38.506544 17930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:38.502080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40679 (local address 127.0.98.1:40991)
0927 02:42:38.502202 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:38.502206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:38.502436 (+ 230us) server_negotiation.cc:407] Connection header received
0927 02:42:38.502496 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:38.502499 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:38.502544 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:38.502607 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:38.502859 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:38.503583 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:38.505203 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:38.505889 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:38.505982 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:38.506047 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:38.506335 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 02:42:38.506420 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 02:42:38.506544 17932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:38.502049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39967 (local address 127.0.98.2:37559)
0927 02:42:38.502165 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:38.502169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:38.502278 (+ 109us) server_negotiation.cc:407] Connection header received
0927 02:42:38.502392 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:38.502395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:38.502446 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:38.502524 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:38.502776 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:38.503584 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:38.505203 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:38.505860 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:38.505982 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:38.506047 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:38.506342 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 02:42:38.506419 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 02:42:38.506543 17943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:38.502011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40991 (local address 127.0.98.3:40679)
0927 02:42:38.502346 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:38.502358 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:38.502435 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:38.502623 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:38.502625 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:38.502639 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:38.502728 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:38.502734 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:38.503695 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:38.503699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:38.505057 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:38.505068 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:38.505999 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:38.506003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:38.506147 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:38.506189 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 02:42:38.506281 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 02:42:38.506378 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":151,"thread_start_us":128,"threads_started":1}
I20260927 02:42:38.507277 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12dda33bdb59404cb6fcb4829ddd99ea" candidate_uuid: "73535a3974814af1a054df0689153faf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f40daf5e697343ca91f724ea16a61fb5" is_pre_election: true
I20260927 02:42:38.507273 17603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12dda33bdb59404cb6fcb4829ddd99ea" candidate_uuid: "73535a3974814af1a054df0689153faf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "047b1cf367ea48838147509edb136791" is_pre_election: true
I20260927 02:42:38.507539 17603 raft_consensus.cc:2454] T 12dda33bdb59404cb6fcb4829ddd99ea P 047b1cf367ea48838147509edb136791 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73535a3974814af1a054df0689153faf in term 0.
I20260927 02:42:38.507546 17737 raft_consensus.cc:2454] T 12dda33bdb59404cb6fcb4829ddd99ea P f40daf5e697343ca91f724ea16a61fb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73535a3974814af1a054df0689153faf in term 0.
I20260927 02:42:38.507962 17804 leader_election.cc:302] T 12dda33bdb59404cb6fcb4829ddd99ea P 73535a3974814af1a054df0689153faf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 047b1cf367ea48838147509edb136791, 73535a3974814af1a054df0689153faf; no voters:
I20260927 02:42:38.508210 17942 raft_consensus.cc:2781] T 12dda33bdb59404cb6fcb4829ddd99ea P 73535a3974814af1a054df0689153faf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:42:38.508291 17942 raft_consensus.cc:482] T 12dda33bdb59404cb6fcb4829ddd99ea P 73535a3974814af1a054df0689153faf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:42:38.508337 17942 raft_consensus.cc:3037] T 12dda33bdb59404cb6fcb4829ddd99ea P 73535a3974814af1a054df0689153faf [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:38.509276 17942 raft_consensus.cc:504] T 12dda33bdb59404cb6fcb4829ddd99ea P 73535a3974814af1a054df0689153faf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047b1cf367ea48838147509edb136791" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40991 } } peers { permanent_uuid: "f40daf5e697343ca91f724ea16a61fb5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37559 } } peers { permanent_uuid: "73535a3974814af1a054df0689153faf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42589 } }
I20260927 02:42:38.509564 17942 leader_election.cc:288] T 12dda33bdb59404cb6fcb4829ddd99ea P 73535a3974814af1a054df0689153faf [CANDIDATE]: Term 1 election: Requested vote from peers 047b1cf367ea48838147509edb136791 (127.0.98.1:40991), f40daf5e697343ca91f724ea16a61fb5 (127.0.98.2:37559)
I20260927 02:42:38.509900 17603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12dda33bdb59404cb6fcb4829ddd99ea" candidate_uuid: "73535a3974814af1a054df0689153faf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "047b1cf367ea48838147509edb136791"
I20260927 02:42:38.509914 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12dda33bdb59404cb6fcb4829ddd99ea" candidate_uuid: "73535a3974814af1a054df0689153faf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f40daf5e697343ca91f724ea16a61fb5"
I20260927 02:42:38.510025 17603 raft_consensus.cc:3037] T 12dda33bdb59404cb6fcb4829ddd99ea P 047b1cf367ea48838147509edb136791 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:38.510025 17737 raft_consensus.cc:3037] T 12dda33bdb59404cb6fcb4829ddd99ea P f40daf5e697343ca91f724ea16a61fb5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:38.511294 17603 raft_consensus.cc:2454] T 12dda33bdb59404cb6fcb4829ddd99ea P 047b1cf367ea48838147509edb136791 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73535a3974814af1a054df0689153faf in term 1.
I20260927 02:42:38.511433 17737 raft_consensus.cc:2454] T 12dda33bdb59404cb6fcb4829ddd99ea P f40daf5e697343ca91f724ea16a61fb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73535a3974814af1a054df0689153faf in term 1.
I20260927 02:42:38.511588 17804 leader_election.cc:302] T 12dda33bdb59404cb6fcb4829ddd99ea P 73535a3974814af1a054df0689153faf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 047b1cf367ea48838147509edb136791, 73535a3974814af1a054df0689153faf; no voters:
I20260927 02:42:38.511830 17942 raft_consensus.cc:2781] T 12dda33bdb59404cb6fcb4829ddd99ea P 73535a3974814af1a054df0689153faf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:38.512063 17942 raft_consensus.cc:686] T 12dda33bdb59404cb6fcb4829ddd99ea P 73535a3974814af1a054df0689153faf [term 1 LEADER]: Becoming Leader. State: Replica: 73535a3974814af1a054df0689153faf, State: Running, Role: LEADER
I20260927 02:42:38.512411 17942 consensus_queue.cc:237] T 12dda33bdb59404cb6fcb4829ddd99ea P 73535a3974814af1a054df0689153faf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047b1cf367ea48838147509edb136791" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40991 } } peers { permanent_uuid: "f40daf5e697343ca91f724ea16a61fb5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37559 } } peers { permanent_uuid: "73535a3974814af1a054df0689153faf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42589 } }
I20260927 02:42:38.515519 17457 catalog_manager.cc:5495] T 12dda33bdb59404cb6fcb4829ddd99ea P 73535a3974814af1a054df0689153faf reported cstate change: term changed from 0 to 1, leader changed from <none> to 73535a3974814af1a054df0689153faf (127.0.98.3). New cstate: current_term: 1 leader_uuid: "73535a3974814af1a054df0689153faf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047b1cf367ea48838147509edb136791" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f40daf5e697343ca91f724ea16a61fb5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73535a3974814af1a054df0689153faf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42589 } health_report { overall_health: HEALTHY } } }
I20260927 02:42:38.560289 17933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:38.556887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33190 (local address 127.0.98.3:42589)
0927 02:42:38.557027 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:38.557031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:38.557042 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:42:38.557112 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:38.557115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:38.557169 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:38.557258 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:42:38.557509 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:38.558003 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:38.558875 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:38.559094 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:38.559192 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:38.559482 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:42:38.559908 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:42:38.560025 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:38.560084 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 02:42:38.560141 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 02:42:38.566447 17603 raft_consensus.cc:1261] T 12dda33bdb59404cb6fcb4829ddd99ea P 047b1cf367ea48838147509edb136791 [term 1 FOLLOWER]: Refusing update from remote peer 73535a3974814af1a054df0689153faf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:42:38.566447 17737 raft_consensus.cc:1261] T 12dda33bdb59404cb6fcb4829ddd99ea P f40daf5e697343ca91f724ea16a61fb5 [term 1 FOLLOWER]: Refusing update from remote peer 73535a3974814af1a054df0689153faf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260927 02:42:38.567214 17918 tablet.cc:2369] T 12dda33bdb59404cb6fcb4829ddd99ea P 73535a3974814af1a054df0689153faf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:42:38.567203 17942 consensus_queue.cc:1035] T 12dda33bdb59404cb6fcb4829ddd99ea P 73535a3974814af1a054df0689153faf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f40daf5e697343ca91f724ea16a61fb5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:42:38.567636 17944 consensus_queue.cc:1035] T 12dda33bdb59404cb6fcb4829ddd99ea P 73535a3974814af1a054df0689153faf [LEADER]: Connected to new peer: Peer: permanent_uuid: "047b1cf367ea48838147509edb136791" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:42:38.577569 17948 mvcc.cc:205] Tried to move back new op lower bound from 7333793622279942144 to 7333793622069559296. Current Snapshot: MvccSnapshot[applied={T|T < 7333793622279942144 or (T in {7333793622279942144})}]
I20260927 02:42:38.583415 17457 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:42:38.583555 17457 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:42:38.585599 17457 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5c610f4916aa403eb1a55ce83f9517e5: Sending DeleteTablet for 3 replicas of tablet 12dda33bdb59404cb6fcb4829ddd99ea
I20260927 02:42:38.586301 17583 tablet_service.cc:1514] Processing DeleteTablet for tablet 12dda33bdb59404cb6fcb4829ddd99ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56704
I20260927 02:42:38.586339 17851 tablet_service.cc:1514] Processing DeleteTablet for tablet 12dda33bdb59404cb6fcb4829ddd99ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33178
I20260927 02:42:38.586755 17956 tablet_replica.cc:330] T 12dda33bdb59404cb6fcb4829ddd99ea P 73535a3974814af1a054df0689153faf: stopping tablet replica
I20260927 02:42:38.586760 17717 tablet_service.cc:1514] Processing DeleteTablet for tablet 12dda33bdb59404cb6fcb4829ddd99ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40654
I20260927 02:42:38.587103 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 17520
I20260927 02:42:38.587462 17956 raft_consensus.cc:2229] T 12dda33bdb59404cb6fcb4829ddd99ea P 73535a3974814af1a054df0689153faf [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:42:38.587920 17956 raft_consensus.cc:2258] T 12dda33bdb59404cb6fcb4829ddd99ea P 73535a3974814af1a054df0689153faf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:38.588227 17958 tablet_replica.cc:330] T 12dda33bdb59404cb6fcb4829ddd99ea P f40daf5e697343ca91f724ea16a61fb5: stopping tablet replica
I20260927 02:42:38.588479 17958 raft_consensus.cc:2229] T 12dda33bdb59404cb6fcb4829ddd99ea P f40daf5e697343ca91f724ea16a61fb5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:42:38.588801 17958 raft_consensus.cc:2258] T 12dda33bdb59404cb6fcb4829ddd99ea P f40daf5e697343ca91f724ea16a61fb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:38.589510 17956 ts_tablet_manager.cc:1905] T 12dda33bdb59404cb6fcb4829ddd99ea P 73535a3974814af1a054df0689153faf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:42:38.590083 17958 ts_tablet_manager.cc:1905] T 12dda33bdb59404cb6fcb4829ddd99ea P f40daf5e697343ca91f724ea16a61fb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:42:38.592196 17956 ts_tablet_manager.cc:1918] T 12dda33bdb59404cb6fcb4829ddd99ea P 73535a3974814af1a054df0689153faf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:42:38.592331 17956 log.cc:1196] T 12dda33bdb59404cb6fcb4829ddd99ea P 73535a3974814af1a054df0689153faf: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/ts-2/wal/wals/12dda33bdb59404cb6fcb4829ddd99ea
I20260927 02:42:38.592514 17958 ts_tablet_manager.cc:1918] T 12dda33bdb59404cb6fcb4829ddd99ea P f40daf5e697343ca91f724ea16a61fb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:42:38.592626 17958 log.cc:1196] T 12dda33bdb59404cb6fcb4829ddd99ea P f40daf5e697343ca91f724ea16a61fb5: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790476718023131-392-0/minicluster-data/ts-1/wal/wals/12dda33bdb59404cb6fcb4829ddd99ea
I20260927 02:42:38.592655 17956 ts_tablet_manager.cc:1939] T 12dda33bdb59404cb6fcb4829ddd99ea P 73535a3974814af1a054df0689153faf: Deleting consensus metadata
I20260927 02:42:38.592921 17958 ts_tablet_manager.cc:1939] T 12dda33bdb59404cb6fcb4829ddd99ea P f40daf5e697343ca91f724ea16a61fb5: Deleting consensus metadata
I20260927 02:42:38.593712 17442 catalog_manager.cc:4841] TS 73535a3974814af1a054df0689153faf (127.0.98.3:42589): tablet 12dda33bdb59404cb6fcb4829ddd99ea (table test-table [id=aef045ec939f46448fc4e44947775802]) successfully deleted
I20260927 02:42:38.594183 17444 catalog_manager.cc:4841] TS f40daf5e697343ca91f724ea16a61fb5 (127.0.98.2:37559): tablet 12dda33bdb59404cb6fcb4829ddd99ea (table test-table [id=aef045ec939f46448fc4e44947775802]) successfully deleted
W20260927 02:42:38.595456 17442 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40991 (error 108)
I20260927 02:42:38.596073 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 17654
I20260927 02:42:38.596729 17931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:38.596121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40991 (local address 127.0.0.1:56720)
0927 02:42:38.596587 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:38.596650 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395}
W20260927 02:42:38.597004 17442 catalog_manager.cc:4584] TS 047b1cf367ea48838147509edb136791 (127.0.98.1:40991): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 12dda33bdb59404cb6fcb4829ddd99ea: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40991: connect: Connection refused (error 111)
I20260927 02:42:38.601976 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 17788
I20260927 02:42:38.606630 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 17426
2026-09-27T02:42:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2236 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T02:42:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:42:38Z Disabled control of system clock
I20260927 02:42:38.633458 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34933
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34933
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 02:42:38.739815 17963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:38.740053 17963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:38.740085 17963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:38.743978 17963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:42:38.744029 17963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:38.744043 17963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:38.744058 17963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:42:38.744071 17963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:42:38.748642 17963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34933
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:42:38.749598 17963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:38.750890 17963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:38.758598 17971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:38.758597 17968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:38.758617 17969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:38.758873 17963 server_base.cc:1034] running on GCE node
I20260927 02:42:38.759397 17963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:38.760367 17963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:38.761531 17963 hybrid_clock.cc:648] HybridClock initialized: now 1790476958761506 us; error 32 us; skew 500 ppm
I20260927 02:42:38.763208 17963 webserver.cc:460] Webserver started at http://127.0.98.62:45313/ using document root <none> and password file <none>
I20260927 02:42:38.763736 17963 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:38.763792 17963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:38.763975 17963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:38.765848 17963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/master-0/data/instance:
uuid: "11bf51032d0a4679800c37092e12359f"
format_stamp: "Formatted at 2026-09-27 02:42:38 on dist-test-slave-zv8z"
server_key: "f51eea86eed4bb194457ce468291d11c"
server_key_iv: "33461c496f09c605da3e10a3bd220e68"
server_key_version: "encryptionkey@0"
I20260927 02:42:38.766387 17963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/master-0/wal/instance:
uuid: "11bf51032d0a4679800c37092e12359f"
format_stamp: "Formatted at 2026-09-27 02:42:38 on dist-test-slave-zv8z"
server_key: "f51eea86eed4bb194457ce468291d11c"
server_key_iv: "33461c496f09c605da3e10a3bd220e68"
server_key_version: "encryptionkey@0"
I20260927 02:42:38.769771 17963 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 02:42:38.772044 17977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:38.773272 17963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:42:38.773384 17963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "11bf51032d0a4679800c37092e12359f"
format_stamp: "Formatted at 2026-09-27 02:42:38 on dist-test-slave-zv8z"
server_key: "f51eea86eed4bb194457ce468291d11c"
server_key_iv: "33461c496f09c605da3e10a3bd220e68"
server_key_version: "encryptionkey@0"
I20260927 02:42:38.773481 17963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:38.788194 17963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:38.788798 17963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:38.788947 17963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:38.795086 17963 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34933
I20260927 02:42:38.795102 18029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34933 every 8 connection(s)
I20260927 02:42:38.795938 17963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:42:38.798118 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 17963
I20260927 02:42:38.798236 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/master-0/wal/instance
I20260927 02:42:38.798524 392 external_mini_cluster.cc:1428] Setting key df34c0acc4fe91336e7de46ca8bbfb36
I20260927 02:42:38.799055 18030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:38.805429 18030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11bf51032d0a4679800c37092e12359f: Bootstrap starting.
I20260927 02:42:38.807188 18033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:38.799804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35834 (local address 127.0.98.62:34933)
0927 02:42:38.800428 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:38.800443 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:38.800477 (+ 34us) server_negotiation.cc:407] Connection header received
0927 02:42:38.801099 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:38.801114 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:38.801497 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:38.801758 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:38.802067 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:38.803039 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:38.804189 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:38.804635 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:38.804743 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:38.805561 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:38.805567 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:38.805578 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:42:38.805956 (+ 378us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:38.806007 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:42:38.806026 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:38.806097 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:38.806244 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 02:42:38.806527 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":104,"threads_started":1}
I20260927 02:42:38.807593 18030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11bf51032d0a4679800c37092e12359f: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:38.808250 18030 log.cc:824] T 00000000000000000000000000000000 P 11bf51032d0a4679800c37092e12359f: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:38.809945 18030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11bf51032d0a4679800c37092e12359f: No bootstrap required, opened a new log
I20260927 02:42:38.813311 18030 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11bf51032d0a4679800c37092e12359f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11bf51032d0a4679800c37092e12359f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34933 } }
I20260927 02:42:38.813511 18030 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11bf51032d0a4679800c37092e12359f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:38.813566 18030 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11bf51032d0a4679800c37092e12359f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11bf51032d0a4679800c37092e12359f, State: Initialized, Role: FOLLOWER
I20260927 02:42:38.813956 18030 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11bf51032d0a4679800c37092e12359f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11bf51032d0a4679800c37092e12359f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34933 } }
I20260927 02:42:38.814079 18030 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11bf51032d0a4679800c37092e12359f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:42:38.814139 18030 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11bf51032d0a4679800c37092e12359f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:42:38.814229 18030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11bf51032d0a4679800c37092e12359f [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:38.815014 18030 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11bf51032d0a4679800c37092e12359f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11bf51032d0a4679800c37092e12359f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34933 } }
I20260927 02:42:38.815310 18030 leader_election.cc:302] T 00000000000000000000000000000000 P 11bf51032d0a4679800c37092e12359f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11bf51032d0a4679800c37092e12359f; no voters:
I20260927 02:42:38.815565 18030 leader_election.cc:288] T 00000000000000000000000000000000 P 11bf51032d0a4679800c37092e12359f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:42:38.815654 18035 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11bf51032d0a4679800c37092e12359f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:38.815843 18035 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11bf51032d0a4679800c37092e12359f [term 1 LEADER]: Becoming Leader. State: Replica: 11bf51032d0a4679800c37092e12359f, State: Running, Role: LEADER
I20260927 02:42:38.816139 18035 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11bf51032d0a4679800c37092e12359f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11bf51032d0a4679800c37092e12359f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34933 } }
I20260927 02:42:38.816614 18030 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11bf51032d0a4679800c37092e12359f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:42:38.817763 18036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11bf51032d0a4679800c37092e12359f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11bf51032d0a4679800c37092e12359f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11bf51032d0a4679800c37092e12359f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34933 } } }
I20260927 02:42:38.817764 18037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11bf51032d0a4679800c37092e12359f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11bf51032d0a4679800c37092e12359f. Latest consensus state: current_term: 1 leader_uuid: "11bf51032d0a4679800c37092e12359f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11bf51032d0a4679800c37092e12359f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34933 } } }
I20260927 02:42:38.817909 18036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11bf51032d0a4679800c37092e12359f [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:38.817911 18037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11bf51032d0a4679800c37092e12359f [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:38.818276 18042 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:42:38.821105 18042 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:42:38.826422 18042 catalog_manager.cc:1347] Generated new cluster ID: a3e8572ac7a84553a78df126f87fe1e8
I20260927 02:42:38.826484 18042 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:42:38.859566 18042 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:42:38.860807 18042 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:42:38.873051 18042 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11bf51032d0a4679800c37092e12359f: Generated new TSK 0
I20260927 02:42:38.873850 18042 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:42:38.882009 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34933
--builtin_ntp_servers=127.0.98.20:37909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 02:42:38.987828 18054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:38.988063 18054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:38.988113 18054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:38.992026 18054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:38.992074 18054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:38.992146 18054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:42:38.996755 18054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:42:38.997728 18054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:38.999082 18054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:39.008172 18062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:39.008185 18059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:39.008188 18060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:39.008240 18054 server_base.cc:1034] running on GCE node
I20260927 02:42:39.008796 18054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:39.009395 18054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:39.010545 18054 hybrid_clock.cc:648] HybridClock initialized: now 1790476959010523 us; error 35 us; skew 500 ppm
I20260927 02:42:39.012241 18054 webserver.cc:460] Webserver started at http://127.0.98.1:33883/ using document root <none> and password file <none>
I20260927 02:42:39.012776 18054 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:39.012842 18054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:39.013032 18054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:39.015059 18054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/ts-0/data/instance:
uuid: "c3d26992c5194753b86d7cd7391b9d3c"
format_stamp: "Formatted at 2026-09-27 02:42:39 on dist-test-slave-zv8z"
server_key: "846db80af51e5280c3ab875adf8c28a4"
server_key_iv: "a6c414d1564825cb2d3080ca3b7a5b00"
server_key_version: "encryptionkey@0"
I20260927 02:42:39.015575 18054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c3d26992c5194753b86d7cd7391b9d3c"
format_stamp: "Formatted at 2026-09-27 02:42:39 on dist-test-slave-zv8z"
server_key: "846db80af51e5280c3ab875adf8c28a4"
server_key_iv: "a6c414d1564825cb2d3080ca3b7a5b00"
server_key_version: "encryptionkey@0"
I20260927 02:42:39.019127 18054 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:42:39.021411 18068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:39.022599 18054 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:42:39.022697 18054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/ts-0/wal
uuid: "c3d26992c5194753b86d7cd7391b9d3c"
format_stamp: "Formatted at 2026-09-27 02:42:39 on dist-test-slave-zv8z"
server_key: "846db80af51e5280c3ab875adf8c28a4"
server_key_iv: "a6c414d1564825cb2d3080ca3b7a5b00"
server_key_version: "encryptionkey@0"
I20260927 02:42:39.022784 18054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:39.046420 18054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:39.047057 18054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:39.047227 18054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:39.047787 18054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:39.048681 18054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:39.048727 18054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:39.048777 18054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:39.048805 18054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:39.056864 18054 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44307
I20260927 02:42:39.056900 18181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44307 every 8 connection(s)
I20260927 02:42:39.057696 18054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:42:39.066844 18033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:39.059541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50777 (local address 127.0.98.62:34933)
0927 02:42:39.059676 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:39.059681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:39.060807 (+ 1126us) server_negotiation.cc:407] Connection header received
0927 02:42:39.062183 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:39.062186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:39.062235 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:39.062351 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:39.063197 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:39.063875 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:39.065065 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:39.065288 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:39.065370 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:39.066378 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:39.066382 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:39.066384 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:42:39.066397 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:39.066417 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:42:39.066425 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:39.066493 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:39.066676 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 02:42:39.066731 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 02:42:39.067649 18184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:39.060027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34933 (local address 127.0.98.1:50777)
0927 02:42:39.060666 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:39.060708 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:39.061972 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:39.062495 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:39.062501 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:39.062582 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:39.063058 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:39.063071 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:39.064022 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:39.064025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:39.064930 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:39.064940 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:39.065409 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:39.065411 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:39.065545 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:39.066069 (+ 524us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:42:39.066070 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:39.066110 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:42:39.066170 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:39.066544 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:39.066552 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:39.066557 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:42:39.066668 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 02:42:39.066898 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":112,"threads_started":1}
I20260927 02:42:39.067880 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 18054
I20260927 02:42:39.067994 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:42:39.068305 392 external_mini_cluster.cc:1428] Setting key ae479220df3478aae981ad70f5a6028e
I20260927 02:42:39.068900 18182 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34933
I20260927 02:42:39.069119 18182 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:39.069666 18182 heartbeater.cc:511] Master 127.0.98.62:34933 requested a full tablet report, sending...
I20260927 02:42:39.070072 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34933
--builtin_ntp_servers=127.0.98.20:37909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 02:42:39.071182 17994 ts_manager.cc:194] Registered new tserver with Master: c3d26992c5194753b86d7cd7391b9d3c (127.0.98.1:44307)
I20260927 02:42:39.072320 17994 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:50777
W20260927 02:42:39.175203 18185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:39.175462 18185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:39.175511 18185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:39.179586 18185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:39.179638 18185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:39.179714 18185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:42:39.184331 18185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:42:39.185333 18185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:39.186726 18185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:39.195928 18193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:39.195868 18191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:39.195860 18190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:39.196135 18185 server_base.cc:1034] running on GCE node
I20260927 02:42:39.196571 18185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:39.197166 18185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:39.198315 18185 hybrid_clock.cc:648] HybridClock initialized: now 1790476959198295 us; error 31 us; skew 500 ppm
I20260927 02:42:39.200039 18185 webserver.cc:460] Webserver started at http://127.0.98.2:37833/ using document root <none> and password file <none>
I20260927 02:42:39.200541 18185 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:39.200594 18185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:39.200749 18185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:39.202713 18185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/ts-1/data/instance:
uuid: "eec08f6906354724b77a36b4429d37a7"
format_stamp: "Formatted at 2026-09-27 02:42:39 on dist-test-slave-zv8z"
server_key: "62bedaaa8e991c735987d72e89d0f1b1"
server_key_iv: "f6b1a9c651c587d55990aa6162ad4a56"
server_key_version: "encryptionkey@0"
I20260927 02:42:39.203189 18185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/ts-1/wal/instance:
uuid: "eec08f6906354724b77a36b4429d37a7"
format_stamp: "Formatted at 2026-09-27 02:42:39 on dist-test-slave-zv8z"
server_key: "62bedaaa8e991c735987d72e89d0f1b1"
server_key_iv: "f6b1a9c651c587d55990aa6162ad4a56"
server_key_version: "encryptionkey@0"
I20260927 02:42:39.206542 18185 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:42:39.208813 18199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:39.209985 18185 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:42:39.210086 18185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/ts-1/wal
uuid: "eec08f6906354724b77a36b4429d37a7"
format_stamp: "Formatted at 2026-09-27 02:42:39 on dist-test-slave-zv8z"
server_key: "62bedaaa8e991c735987d72e89d0f1b1"
server_key_iv: "f6b1a9c651c587d55990aa6162ad4a56"
server_key_version: "encryptionkey@0"
I20260927 02:42:39.210184 18185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:39.223636 18185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:39.224246 18185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:39.224408 18185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:39.224948 18185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:39.225884 18185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:39.225939 18185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:39.225980 18185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:39.226009 18185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:39.234205 18185 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40153
I20260927 02:42:39.234234 18312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40153 every 8 connection(s)
I20260927 02:42:39.235106 18185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:42:39.236018 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 18185
I20260927 02:42:39.236123 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:42:39.236405 392 external_mini_cluster.cc:1428] Setting key 4894f080a4b3365973adfd04a3fadb9b
I20260927 02:42:39.238499 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34933
--builtin_ntp_servers=127.0.98.20:37909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 02:42:39.245330 18033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:39.236974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59059 (local address 127.0.98.62:34933)
0927 02:42:39.237103 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:39.237107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:39.238319 (+ 1212us) server_negotiation.cc:407] Connection header received
0927 02:42:39.240063 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:39.240066 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:39.240122 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:39.240209 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:39.241185 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:39.241698 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:39.243391 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:39.243601 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:39.243675 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:39.244910 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:39.244913 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:39.244915 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:42:39.244927 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:39.244945 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:42:39.244953 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:39.245006 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:39.245170 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 02:42:39.245219 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 02:42:39.246538 18315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:39.237366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34933 (local address 127.0.98.2:59059)
0927 02:42:39.238159 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:39.238213 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:39.239819 (+ 1606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:39.240353 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:39.240360 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:39.240445 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:39.241045 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:39.241064 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:39.241813 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:39.241817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:39.243275 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:39.243287 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:39.243760 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:39.243765 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:39.243983 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:39.244589 (+ 606us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:42:39.244591 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:39.244649 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:42:39.244749 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:39.245030 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:39.245039 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:39.245046 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 02:42:39.245175 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 02:42:39.245481 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":111,"threads_started":1}
I20260927 02:42:39.247689 18313 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34933
I20260927 02:42:39.247936 18313 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:39.248571 18313 heartbeater.cc:511] Master 127.0.98.62:34933 requested a full tablet report, sending...
I20260927 02:42:39.249621 17994 ts_manager.cc:194] Registered new tserver with Master: eec08f6906354724b77a36b4429d37a7 (127.0.98.2:40153)
I20260927 02:42:39.250177 17994 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:59059
W20260927 02:42:39.346977 18316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:39.347251 18316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:39.347291 18316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:39.351287 18316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:39.351347 18316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:39.351401 18316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:42:39.356221 18316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:42:39.357254 18316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:39.358601 18316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:39.368386 18322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:39.368448 18324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:39.368448 18316 server_base.cc:1034] running on GCE node
W20260927 02:42:39.368376 18321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:39.368997 18316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:39.369580 18316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:39.370733 18316 hybrid_clock.cc:648] HybridClock initialized: now 1790476959370722 us; error 26 us; skew 500 ppm
I20260927 02:42:39.372390 18316 webserver.cc:460] Webserver started at http://127.0.98.3:32917/ using document root <none> and password file <none>
I20260927 02:42:39.372928 18316 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:39.372990 18316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:39.373162 18316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:39.375180 18316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/ts-2/data/instance:
uuid: "12fabdbd4a484363bd559a2e333984e1"
format_stamp: "Formatted at 2026-09-27 02:42:39 on dist-test-slave-zv8z"
server_key: "5e149aef940a4169a454beb5a8092646"
server_key_iv: "33f5cc421120b6b60c88280ff82de03c"
server_key_version: "encryptionkey@0"
I20260927 02:42:39.375682 18316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/ts-2/wal/instance:
uuid: "12fabdbd4a484363bd559a2e333984e1"
format_stamp: "Formatted at 2026-09-27 02:42:39 on dist-test-slave-zv8z"
server_key: "5e149aef940a4169a454beb5a8092646"
server_key_iv: "33f5cc421120b6b60c88280ff82de03c"
server_key_version: "encryptionkey@0"
I20260927 02:42:39.379184 18316 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:42:39.381405 18330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:39.382520 18316 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:42:39.382616 18316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/ts-2/wal
uuid: "12fabdbd4a484363bd559a2e333984e1"
format_stamp: "Formatted at 2026-09-27 02:42:39 on dist-test-slave-zv8z"
server_key: "5e149aef940a4169a454beb5a8092646"
server_key_iv: "33f5cc421120b6b60c88280ff82de03c"
server_key_version: "encryptionkey@0"
I20260927 02:42:39.382701 18316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:39.407419 18316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:39.408061 18316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:39.408221 18316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:39.408756 18316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:39.409636 18316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:39.409682 18316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:39.409734 18316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:39.409762 18316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:39.418105 18316 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42967
I20260927 02:42:39.418145 18443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42967 every 8 connection(s)
I20260927 02:42:39.418962 18316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:42:39.424665 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 18316
I20260927 02:42:39.424760 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:42:39.425007 392 external_mini_cluster.cc:1428] Setting key 743eb0c5be206b438e7e949f82230c6c
I20260927 02:42:39.427791 18033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:39.420659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49143 (local address 127.0.98.62:34933)
0927 02:42:39.420790 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:39.420794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:39.421830 (+ 1036us) server_negotiation.cc:407] Connection header received
0927 02:42:39.423230 (+ 1400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:39.423233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:39.423288 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:39.423362 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:39.424270 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:39.424846 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:39.426009 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:39.426244 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:39.426360 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:39.427385 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:39.427388 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:39.427390 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:42:39.427401 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:39.427422 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:42:39.427429 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:39.427484 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:39.427620 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 02:42:39.427677 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 02:42:39.428567 18446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:39.421060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34933 (local address 127.0.98.3:49143)
0927 02:42:39.421687 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:39.421726 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:39.423004 (+ 1278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:39.423537 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:39.423544 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:39.423629 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:39.424135 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:39.424147 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:39.424961 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:39.424964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:39.425906 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:39.425916 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:39.426391 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:39.426395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:39.426552 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:39.427154 (+ 602us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:42:39.427156 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:39.427194 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:42:39.427254 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:39.427485 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:39.427490 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:39.427493 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:42:39.427575 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 02:42:39.427788 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":103,"threads_started":1}
I20260927 02:42:39.429539 18444 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34933
I20260927 02:42:39.429762 18444 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:39.430182 18444 heartbeater.cc:511] Master 127.0.98.62:34933 requested a full tablet report, sending...
I20260927 02:42:39.431079 17994 ts_manager.cc:194] Registered new tserver with Master: 12fabdbd4a484363bd559a2e333984e1 (127.0.98.3:42967)
I20260927 02:42:39.431604 17994 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:49143
I20260927 02:42:39.438369 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:42:39.443655 18033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:39.440163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35850 (local address 127.0.98.62:34933)
0927 02:42:39.440279 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:39.440283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:39.440477 (+ 194us) server_negotiation.cc:407] Connection header received
0927 02:42:39.440582 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:39.440585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:39.440627 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:39.440689 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:39.440946 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:39.441433 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:39.442205 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:39.442487 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:39.442571 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:39.443262 (+ 691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:39.443266 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:39.443268 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:42:39.443280 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:39.443300 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:42:39.443309 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:39.443365 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:39.443478 (+ 113us) server_negotiation.cc:299] Negotiation successful
0927 02:42:39.443526 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 02:42:39.446033 17994 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35850:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:42:39.448881 17994 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:42:39.462119 18456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:39.456443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40153 (local address 127.0.0.1:47354)
0927 02:42:39.457123 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:39.457164 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:39.457293 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:39.457973 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:39.457979 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:39.458015 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:39.458213 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:39.458220 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:39.459631 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:39.459634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:39.460779 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:39.460788 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:39.461709 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:39.461712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:39.461795 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:39.461823 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 02:42:39.461894 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 02:42:39.461973 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"mutex_wait_us":268,"spinlock_wait_cycles":6272,"thread_start_us":84,"threads_started":1}
I20260927 02:42:39.462179 18457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:39.456585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47354 (local address 127.0.98.2:40153)
0927 02:42:39.457494 (+ 909us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:39.457505 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:39.457524 (+ 19us) server_negotiation.cc:407] Connection header received
0927 02:42:39.457586 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:39.457590 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:39.457741 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:39.457854 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:39.458361 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:39.459534 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:39.460891 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:39.461593 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:39.461698 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:39.461880 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:39.461933 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 02:42:39.462025 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":812,"thread_start_us":133,"threads_started":1}
I20260927 02:42:39.462482 18458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:39.456654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42946 (local address 127.0.98.3:42967)
0927 02:42:39.457260 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:39.457280 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:39.457309 (+ 29us) server_negotiation.cc:407] Connection header received
0927 02:42:39.457463 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:39.457468 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:39.457669 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:39.457811 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:39.458362 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:39.459534 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:39.461178 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:39.461934 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:39.462028 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:39.462192 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:39.462242 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 02:42:39.462361 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":123,"threads_started":1}
I20260927 02:42:39.462517 18455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:39.456443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42967 (local address 127.0.0.1:42946)
0927 02:42:39.457107 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:39.457157 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:39.457295 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:39.457973 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:39.457979 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:39.458013 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:39.458213 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:39.458220 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:39.459688 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:39.459692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:39.461057 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:39.461068 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:39.462056 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:39.462059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:39.462136 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:39.462159 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 02:42:39.462217 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 02:42:39.462394 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"mutex_wait_us":139,"spinlock_wait_cycles":17536,"thread_start_us":65,"threads_started":1}
I20260927 02:42:39.464119 18459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:39.457157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44307 (local address 127.0.0.1:40284)
0927 02:42:39.458001 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:39.458012 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:39.458098 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:39.458513 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:39.458516 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:39.458532 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:39.458620 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:39.458626 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:39.459541 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:39.459544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:39.461057 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:39.461068 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:39.463740 (+ 2672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:39.463743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:39.463839 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:39.463861 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 02:42:39.463926 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 02:42:39.464000 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"mutex_wait_us":160,"thread_start_us":81,"threads_started":1}
I20260927 02:42:39.464558 18460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:39.457622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40284 (local address 127.0.98.1:44307)
0927 02:42:39.457957 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:39.457969 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:39.458092 (+ 123us) server_negotiation.cc:407] Connection header received
0927 02:42:39.458209 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:39.458213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:39.458402 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:39.458529 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:39.458705 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:39.459442 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:39.462762 (+ 3320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:39.463425 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:39.463518 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:39.463682 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:39.464325 (+ 643us) server_negotiation.cc:299] Negotiation successful
0927 02:42:39.464417 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":128,"threads_started":1}
I20260927 02:42:39.465443 18378 tablet_service.cc:1467] Processing CreateTablet for tablet 21d8189c4a074a38826cd19bc5b99bbd (DEFAULT_TABLE table=test-table [id=2da9bc34d3414cee96b58c08845dae39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:39.465421 18247 tablet_service.cc:1467] Processing CreateTablet for tablet 21d8189c4a074a38826cd19bc5b99bbd (DEFAULT_TABLE table=test-table [id=2da9bc34d3414cee96b58c08845dae39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:39.466336 18247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21d8189c4a074a38826cd19bc5b99bbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:39.466367 18378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21d8189c4a074a38826cd19bc5b99bbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:39.468351 18116 tablet_service.cc:1467] Processing CreateTablet for tablet 21d8189c4a074a38826cd19bc5b99bbd (DEFAULT_TABLE table=test-table [id=2da9bc34d3414cee96b58c08845dae39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:39.469337 18116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21d8189c4a074a38826cd19bc5b99bbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:39.472160 18461 tablet_bootstrap.cc:492] T 21d8189c4a074a38826cd19bc5b99bbd P 12fabdbd4a484363bd559a2e333984e1: Bootstrap starting.
I20260927 02:42:39.473555 18462 tablet_bootstrap.cc:492] T 21d8189c4a074a38826cd19bc5b99bbd P eec08f6906354724b77a36b4429d37a7: Bootstrap starting.
I20260927 02:42:39.474344 18461 tablet_bootstrap.cc:654] T 21d8189c4a074a38826cd19bc5b99bbd P 12fabdbd4a484363bd559a2e333984e1: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:39.475373 18461 log.cc:824] T 21d8189c4a074a38826cd19bc5b99bbd P 12fabdbd4a484363bd559a2e333984e1: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:39.475592 18462 tablet_bootstrap.cc:654] T 21d8189c4a074a38826cd19bc5b99bbd P eec08f6906354724b77a36b4429d37a7: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:39.476032 18463 tablet_bootstrap.cc:492] T 21d8189c4a074a38826cd19bc5b99bbd P c3d26992c5194753b86d7cd7391b9d3c: Bootstrap starting.
I20260927 02:42:39.476372 18462 log.cc:824] T 21d8189c4a074a38826cd19bc5b99bbd P eec08f6906354724b77a36b4429d37a7: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:39.477741 18463 tablet_bootstrap.cc:654] T 21d8189c4a074a38826cd19bc5b99bbd P c3d26992c5194753b86d7cd7391b9d3c: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:39.478127 18462 tablet_bootstrap.cc:492] T 21d8189c4a074a38826cd19bc5b99bbd P eec08f6906354724b77a36b4429d37a7: No bootstrap required, opened a new log
I20260927 02:42:39.478325 18462 ts_tablet_manager.cc:1397] T 21d8189c4a074a38826cd19bc5b99bbd P eec08f6906354724b77a36b4429d37a7: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260927 02:42:39.478400 18463 log.cc:824] T 21d8189c4a074a38826cd19bc5b99bbd P c3d26992c5194753b86d7cd7391b9d3c: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:39.479880 18463 tablet_bootstrap.cc:492] T 21d8189c4a074a38826cd19bc5b99bbd P c3d26992c5194753b86d7cd7391b9d3c: No bootstrap required, opened a new log
I20260927 02:42:39.479887 18461 tablet_bootstrap.cc:492] T 21d8189c4a074a38826cd19bc5b99bbd P 12fabdbd4a484363bd559a2e333984e1: No bootstrap required, opened a new log
I20260927 02:42:39.480044 18463 ts_tablet_manager.cc:1397] T 21d8189c4a074a38826cd19bc5b99bbd P c3d26992c5194753b86d7cd7391b9d3c: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260927 02:42:39.480046 18461 ts_tablet_manager.cc:1397] T 21d8189c4a074a38826cd19bc5b99bbd P 12fabdbd4a484363bd559a2e333984e1: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260927 02:42:39.482370 18462 raft_consensus.cc:348] T 21d8189c4a074a38826cd19bc5b99bbd P eec08f6906354724b77a36b4429d37a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eec08f6906354724b77a36b4429d37a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40153 } } peers { permanent_uuid: "12fabdbd4a484363bd559a2e333984e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42967 } } peers { permanent_uuid: "c3d26992c5194753b86d7cd7391b9d3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44307 } }
I20260927 02:42:39.482717 18462 raft_consensus.cc:374] T 21d8189c4a074a38826cd19bc5b99bbd P eec08f6906354724b77a36b4429d37a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:39.482818 18462 raft_consensus.cc:729] T 21d8189c4a074a38826cd19bc5b99bbd P eec08f6906354724b77a36b4429d37a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eec08f6906354724b77a36b4429d37a7, State: Initialized, Role: FOLLOWER
I20260927 02:42:39.483160 18463 raft_consensus.cc:348] T 21d8189c4a074a38826cd19bc5b99bbd P c3d26992c5194753b86d7cd7391b9d3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eec08f6906354724b77a36b4429d37a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40153 } } peers { permanent_uuid: "12fabdbd4a484363bd559a2e333984e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42967 } } peers { permanent_uuid: "c3d26992c5194753b86d7cd7391b9d3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44307 } }
I20260927 02:42:39.483354 18463 raft_consensus.cc:374] T 21d8189c4a074a38826cd19bc5b99bbd P c3d26992c5194753b86d7cd7391b9d3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:39.483284 18462 consensus_queue.cc:260] T 21d8189c4a074a38826cd19bc5b99bbd P eec08f6906354724b77a36b4429d37a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eec08f6906354724b77a36b4429d37a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40153 } } peers { permanent_uuid: "12fabdbd4a484363bd559a2e333984e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42967 } } peers { permanent_uuid: "c3d26992c5194753b86d7cd7391b9d3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44307 } }
I20260927 02:42:39.483425 18463 raft_consensus.cc:729] T 21d8189c4a074a38826cd19bc5b99bbd P c3d26992c5194753b86d7cd7391b9d3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d26992c5194753b86d7cd7391b9d3c, State: Initialized, Role: FOLLOWER
I20260927 02:42:39.483871 18463 consensus_queue.cc:260] T 21d8189c4a074a38826cd19bc5b99bbd P c3d26992c5194753b86d7cd7391b9d3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eec08f6906354724b77a36b4429d37a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40153 } } peers { permanent_uuid: "12fabdbd4a484363bd559a2e333984e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42967 } } peers { permanent_uuid: "c3d26992c5194753b86d7cd7391b9d3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44307 } }
I20260927 02:42:39.483927 18313 heartbeater.cc:503] Master 127.0.98.62:34933 was elected leader, sending a full tablet report...
I20260927 02:42:39.484226 18462 ts_tablet_manager.cc:1428] T 21d8189c4a074a38826cd19bc5b99bbd P eec08f6906354724b77a36b4429d37a7: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 02:42:39.484412 18182 heartbeater.cc:503] Master 127.0.98.62:34933 was elected leader, sending a full tablet report...
I20260927 02:42:39.485191 18461 raft_consensus.cc:348] T 21d8189c4a074a38826cd19bc5b99bbd P 12fabdbd4a484363bd559a2e333984e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eec08f6906354724b77a36b4429d37a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40153 } } peers { permanent_uuid: "12fabdbd4a484363bd559a2e333984e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42967 } } peers { permanent_uuid: "c3d26992c5194753b86d7cd7391b9d3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44307 } }
I20260927 02:42:39.485476 18461 raft_consensus.cc:374] T 21d8189c4a074a38826cd19bc5b99bbd P 12fabdbd4a484363bd559a2e333984e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:39.485558 18463 ts_tablet_manager.cc:1428] T 21d8189c4a074a38826cd19bc5b99bbd P c3d26992c5194753b86d7cd7391b9d3c: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260927 02:42:39.485565 18461 raft_consensus.cc:729] T 21d8189c4a074a38826cd19bc5b99bbd P 12fabdbd4a484363bd559a2e333984e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12fabdbd4a484363bd559a2e333984e1, State: Initialized, Role: FOLLOWER
W20260927 02:42:39.486017 18314 tablet.cc:2369] T 21d8189c4a074a38826cd19bc5b99bbd P eec08f6906354724b77a36b4429d37a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:42:39.486047 18461 consensus_queue.cc:260] T 21d8189c4a074a38826cd19bc5b99bbd P 12fabdbd4a484363bd559a2e333984e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eec08f6906354724b77a36b4429d37a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40153 } } peers { permanent_uuid: "12fabdbd4a484363bd559a2e333984e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42967 } } peers { permanent_uuid: "c3d26992c5194753b86d7cd7391b9d3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44307 } }
I20260927 02:42:39.486860 18444 heartbeater.cc:503] Master 127.0.98.62:34933 was elected leader, sending a full tablet report...
I20260927 02:42:39.486912 18461 ts_tablet_manager.cc:1428] T 21d8189c4a074a38826cd19bc5b99bbd P 12fabdbd4a484363bd559a2e333984e1: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
W20260927 02:42:39.558794 18183 tablet.cc:2369] T 21d8189c4a074a38826cd19bc5b99bbd P c3d26992c5194753b86d7cd7391b9d3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:42:39.669865 18445 tablet.cc:2369] T 21d8189c4a074a38826cd19bc5b99bbd P 12fabdbd4a484363bd559a2e333984e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:42:39.821358 18467 raft_consensus.cc:482] T 21d8189c4a074a38826cd19bc5b99bbd P eec08f6906354724b77a36b4429d37a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:42:39.821554 18467 raft_consensus.cc:504] T 21d8189c4a074a38826cd19bc5b99bbd P eec08f6906354724b77a36b4429d37a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eec08f6906354724b77a36b4429d37a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40153 } } peers { permanent_uuid: "12fabdbd4a484363bd559a2e333984e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42967 } } peers { permanent_uuid: "c3d26992c5194753b86d7cd7391b9d3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44307 } }
I20260927 02:42:39.822700 18467 leader_election.cc:288] T 21d8189c4a074a38826cd19bc5b99bbd P eec08f6906354724b77a36b4429d37a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12fabdbd4a484363bd559a2e333984e1 (127.0.98.3:42967), c3d26992c5194753b86d7cd7391b9d3c (127.0.98.1:44307)
I20260927 02:42:39.827689 18458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:39.823050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52413 (local address 127.0.98.3:42967)
0927 02:42:39.823176 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:39.823180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:39.823302 (+ 122us) server_negotiation.cc:407] Connection header received
0927 02:42:39.823441 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:39.823444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:39.823496 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:39.823581 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:39.823925 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:39.824714 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:39.826382 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:39.827042 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:39.827129 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:39.827178 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:39.827487 (+ 309us) server_negotiation.cc:299] Negotiation successful
0927 02:42:39.827577 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 02:42:39.827697 18460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:39.823047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56633 (local address 127.0.98.1:44307)
0927 02:42:39.823176 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:39.823180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:39.823350 (+ 170us) server_negotiation.cc:407] Connection header received
0927 02:42:39.823441 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:39.823444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:39.823500 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:39.823575 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:39.823925 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:39.824703 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:39.826381 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:39.827009 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:39.827117 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:39.827199 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:39.827488 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 02:42:39.827566 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 02:42:39.827705 18471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:39.822901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44307 (local address 127.0.98.2:56633)
0927 02:42:39.823263 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:39.823274 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:39.823361 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:39.823601 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:39.823604 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:39.823644 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:39.823804 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:39.823811 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:39.824810 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:39.824813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:39.826226 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:39.826238 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:39.827129 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:39.827134 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:39.827303 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:39.827351 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 02:42:39.827430 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 02:42:39.827518 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":135,"spinlock_wait_cycles":32384,"thread_start_us":60,"threads_started":1}
I20260927 02:42:39.827703 18470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:39.822901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42967 (local address 127.0.98.2:52413)
0927 02:42:39.823216 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:39.823231 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:39.823348 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:39.823601 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:39.823604 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:39.823646 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:39.823796 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:39.823838 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:39.824840 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:39.824843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:39.826234 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:39.826240 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:39.827149 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:39.827152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:39.827288 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:39.827347 (+ 59us) client_negotiation.cc:769] Sending connection context
0927 02:42:39.827430 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 02:42:39.827518 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":145,"spinlock_wait_cycles":2432,"thread_start_us":84,"threads_started":1}
I20260927 02:42:39.828398 18136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21d8189c4a074a38826cd19bc5b99bbd" candidate_uuid: "eec08f6906354724b77a36b4429d37a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26992c5194753b86d7cd7391b9d3c" is_pre_election: true
I20260927 02:42:39.828630 18398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21d8189c4a074a38826cd19bc5b99bbd" candidate_uuid: "eec08f6906354724b77a36b4429d37a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12fabdbd4a484363bd559a2e333984e1" is_pre_election: true
I20260927 02:42:39.828677 18136 raft_consensus.cc:2454] T 21d8189c4a074a38826cd19bc5b99bbd P c3d26992c5194753b86d7cd7391b9d3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eec08f6906354724b77a36b4429d37a7 in term 0.
I20260927 02:42:39.828888 18398 raft_consensus.cc:2454] T 21d8189c4a074a38826cd19bc5b99bbd P 12fabdbd4a484363bd559a2e333984e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eec08f6906354724b77a36b4429d37a7 in term 0.
I20260927 02:42:39.829118 18202 leader_election.cc:302] T 21d8189c4a074a38826cd19bc5b99bbd P eec08f6906354724b77a36b4429d37a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3d26992c5194753b86d7cd7391b9d3c, eec08f6906354724b77a36b4429d37a7; no voters:
I20260927 02:42:39.829401 18467 raft_consensus.cc:2781] T 21d8189c4a074a38826cd19bc5b99bbd P eec08f6906354724b77a36b4429d37a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:42:39.829483 18467 raft_consensus.cc:482] T 21d8189c4a074a38826cd19bc5b99bbd P eec08f6906354724b77a36b4429d37a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:42:39.829530 18467 raft_consensus.cc:3037] T 21d8189c4a074a38826cd19bc5b99bbd P eec08f6906354724b77a36b4429d37a7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:39.830489 18467 raft_consensus.cc:504] T 21d8189c4a074a38826cd19bc5b99bbd P eec08f6906354724b77a36b4429d37a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eec08f6906354724b77a36b4429d37a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40153 } } peers { permanent_uuid: "12fabdbd4a484363bd559a2e333984e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42967 } } peers { permanent_uuid: "c3d26992c5194753b86d7cd7391b9d3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44307 } }
I20260927 02:42:39.830782 18467 leader_election.cc:288] T 21d8189c4a074a38826cd19bc5b99bbd P eec08f6906354724b77a36b4429d37a7 [CANDIDATE]: Term 1 election: Requested vote from peers 12fabdbd4a484363bd559a2e333984e1 (127.0.98.3:42967), c3d26992c5194753b86d7cd7391b9d3c (127.0.98.1:44307)
I20260927 02:42:39.831108 18398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21d8189c4a074a38826cd19bc5b99bbd" candidate_uuid: "eec08f6906354724b77a36b4429d37a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12fabdbd4a484363bd559a2e333984e1"
I20260927 02:42:39.831136 18136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21d8189c4a074a38826cd19bc5b99bbd" candidate_uuid: "eec08f6906354724b77a36b4429d37a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3d26992c5194753b86d7cd7391b9d3c"
I20260927 02:42:39.831228 18136 raft_consensus.cc:3037] T 21d8189c4a074a38826cd19bc5b99bbd P c3d26992c5194753b86d7cd7391b9d3c [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:39.831240 18398 raft_consensus.cc:3037] T 21d8189c4a074a38826cd19bc5b99bbd P 12fabdbd4a484363bd559a2e333984e1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:39.832130 18136 raft_consensus.cc:2454] T 21d8189c4a074a38826cd19bc5b99bbd P c3d26992c5194753b86d7cd7391b9d3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eec08f6906354724b77a36b4429d37a7 in term 1.
I20260927 02:42:39.832229 18398 raft_consensus.cc:2454] T 21d8189c4a074a38826cd19bc5b99bbd P 12fabdbd4a484363bd559a2e333984e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eec08f6906354724b77a36b4429d37a7 in term 1.
I20260927 02:42:39.832460 18202 leader_election.cc:302] T 21d8189c4a074a38826cd19bc5b99bbd P eec08f6906354724b77a36b4429d37a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3d26992c5194753b86d7cd7391b9d3c, eec08f6906354724b77a36b4429d37a7; no voters:
I20260927 02:42:39.832620 18467 raft_consensus.cc:2781] T 21d8189c4a074a38826cd19bc5b99bbd P eec08f6906354724b77a36b4429d37a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:39.832835 18467 raft_consensus.cc:686] T 21d8189c4a074a38826cd19bc5b99bbd P eec08f6906354724b77a36b4429d37a7 [term 1 LEADER]: Becoming Leader. State: Replica: eec08f6906354724b77a36b4429d37a7, State: Running, Role: LEADER
I20260927 02:42:39.833153 18467 consensus_queue.cc:237] T 21d8189c4a074a38826cd19bc5b99bbd P eec08f6906354724b77a36b4429d37a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eec08f6906354724b77a36b4429d37a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40153 } } peers { permanent_uuid: "12fabdbd4a484363bd559a2e333984e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42967 } } peers { permanent_uuid: "c3d26992c5194753b86d7cd7391b9d3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44307 } }
I20260927 02:42:39.836064 17992 catalog_manager.cc:5495] T 21d8189c4a074a38826cd19bc5b99bbd P eec08f6906354724b77a36b4429d37a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to eec08f6906354724b77a36b4429d37a7 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "eec08f6906354724b77a36b4429d37a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eec08f6906354724b77a36b4429d37a7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12fabdbd4a484363bd559a2e333984e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3d26992c5194753b86d7cd7391b9d3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44307 } health_report { overall_health: UNKNOWN } } }
I20260927 02:42:39.914919 18457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:39.911536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47370 (local address 127.0.98.2:40153)
0927 02:42:39.911660 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:39.911663 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:39.911676 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:42:39.911773 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:39.911776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:39.911856 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:39.911942 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:42:39.912137 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:39.912621 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:39.913482 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:39.913704 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:39.913773 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:39.914103 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:42:39.914548 (+ 445us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:42:39.914633 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:39.914710 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 02:42:39.914754 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 02:42:39.920571 18136 raft_consensus.cc:1261] T 21d8189c4a074a38826cd19bc5b99bbd P c3d26992c5194753b86d7cd7391b9d3c [term 1 FOLLOWER]: Refusing update from remote peer eec08f6906354724b77a36b4429d37a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:42:39.920565 18398 raft_consensus.cc:1261] T 21d8189c4a074a38826cd19bc5b99bbd P 12fabdbd4a484363bd559a2e333984e1 [term 1 FOLLOWER]: Refusing update from remote peer eec08f6906354724b77a36b4429d37a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:42:39.921236 18467 consensus_queue.cc:1035] T 21d8189c4a074a38826cd19bc5b99bbd P eec08f6906354724b77a36b4429d37a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12fabdbd4a484363bd559a2e333984e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:42:39.921413 18472 consensus_queue.cc:1035] T 21d8189c4a074a38826cd19bc5b99bbd P eec08f6906354724b77a36b4429d37a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d26992c5194753b86d7cd7391b9d3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:42:39.929183 18476 mvcc.cc:205] Tried to move back new op lower bound from 7333793627827933184 to 7333793627478896640. Current Snapshot: MvccSnapshot[applied={T|T < 7333793627827933184}]
I20260927 02:42:39.929419 18475 mvcc.cc:205] Tried to move back new op lower bound from 7333793627827933184 to 7333793627478896640. Current Snapshot: MvccSnapshot[applied={T|T < 7333793627827933184}]
I20260927 02:42:39.930318 18477 mvcc.cc:205] Tried to move back new op lower bound from 7333793627827933184 to 7333793627478896640. Current Snapshot: MvccSnapshot[applied={T|T < 7333793627827933184}]
I20260927 02:42:39.937601 17994 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:42:39.937779 17994 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:42:39.940114 17994 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 11bf51032d0a4679800c37092e12359f: Sending DeleteTablet for 3 replicas of tablet 21d8189c4a074a38826cd19bc5b99bbd
I20260927 02:42:39.940785 18247 tablet_service.cc:1514] Processing DeleteTablet for tablet 21d8189c4a074a38826cd19bc5b99bbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:39 UTC) from {username='slave'} at 127.0.0.1:47354
I20260927 02:42:39.940783 18378 tablet_service.cc:1514] Processing DeleteTablet for tablet 21d8189c4a074a38826cd19bc5b99bbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:39 UTC) from {username='slave'} at 127.0.0.1:42946
I20260927 02:42:39.941004 18116 tablet_service.cc:1514] Processing DeleteTablet for tablet 21d8189c4a074a38826cd19bc5b99bbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:39 UTC) from {username='slave'} at 127.0.0.1:40284
I20260927 02:42:39.941273 18485 tablet_replica.cc:330] T 21d8189c4a074a38826cd19bc5b99bbd P eec08f6906354724b77a36b4429d37a7: stopping tablet replica
I20260927 02:42:39.941449 18486 tablet_replica.cc:330] T 21d8189c4a074a38826cd19bc5b99bbd P c3d26992c5194753b86d7cd7391b9d3c: stopping tablet replica
I20260927 02:42:39.941663 18485 raft_consensus.cc:2229] T 21d8189c4a074a38826cd19bc5b99bbd P eec08f6906354724b77a36b4429d37a7 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:42:39.941679 18486 raft_consensus.cc:2229] T 21d8189c4a074a38826cd19bc5b99bbd P c3d26992c5194753b86d7cd7391b9d3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:42:39.941987 18486 raft_consensus.cc:2258] T 21d8189c4a074a38826cd19bc5b99bbd P c3d26992c5194753b86d7cd7391b9d3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:39.942071 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 18054
I20260927 02:42:39.942723 18485 raft_consensus.cc:2258] T 21d8189c4a074a38826cd19bc5b99bbd P eec08f6906354724b77a36b4429d37a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:39.943372 18484 tablet_replica.cc:330] T 21d8189c4a074a38826cd19bc5b99bbd P 12fabdbd4a484363bd559a2e333984e1: stopping tablet replica
I20260927 02:42:39.943686 18484 raft_consensus.cc:2229] T 21d8189c4a074a38826cd19bc5b99bbd P 12fabdbd4a484363bd559a2e333984e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:42:39.943988 18485 ts_tablet_manager.cc:1905] T 21d8189c4a074a38826cd19bc5b99bbd P eec08f6906354724b77a36b4429d37a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:42:39.944211 18484 raft_consensus.cc:2258] T 21d8189c4a074a38826cd19bc5b99bbd P 12fabdbd4a484363bd559a2e333984e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:39.945612 18484 ts_tablet_manager.cc:1905] T 21d8189c4a074a38826cd19bc5b99bbd P 12fabdbd4a484363bd559a2e333984e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:42:39.946714 18485 ts_tablet_manager.cc:1918] T 21d8189c4a074a38826cd19bc5b99bbd P eec08f6906354724b77a36b4429d37a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:42:39.946825 18485 log.cc:1196] T 21d8189c4a074a38826cd19bc5b99bbd P eec08f6906354724b77a36b4429d37a7: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/ts-1/wal/wals/21d8189c4a074a38826cd19bc5b99bbd
I20260927 02:42:39.947113 18485 ts_tablet_manager.cc:1939] T 21d8189c4a074a38826cd19bc5b99bbd P eec08f6906354724b77a36b4429d37a7: Deleting consensus metadata
I20260927 02:42:39.948278 17978 catalog_manager.cc:4841] TS eec08f6906354724b77a36b4429d37a7 (127.0.98.2:40153): tablet 21d8189c4a074a38826cd19bc5b99bbd (table test-table [id=2da9bc34d3414cee96b58c08845dae39]) successfully deleted
I20260927 02:42:39.948345 18484 ts_tablet_manager.cc:1918] T 21d8189c4a074a38826cd19bc5b99bbd P 12fabdbd4a484363bd559a2e333984e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:42:39.948532 18484 log.cc:1196] T 21d8189c4a074a38826cd19bc5b99bbd P 12fabdbd4a484363bd559a2e333984e1: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790476718023131-392-0/minicluster-data/ts-2/wal/wals/21d8189c4a074a38826cd19bc5b99bbd
I20260927 02:42:39.948791 18484 ts_tablet_manager.cc:1939] T 21d8189c4a074a38826cd19bc5b99bbd P 12fabdbd4a484363bd559a2e333984e1: Deleting consensus metadata
W20260927 02:42:39.949455 17980 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44307 (error 108)
I20260927 02:42:39.949841 17979 catalog_manager.cc:4841] TS 12fabdbd4a484363bd559a2e333984e1 (127.0.98.3:42967): tablet 21d8189c4a074a38826cd19bc5b99bbd (table test-table [id=2da9bc34d3414cee96b58c08845dae39]) successfully deleted
I20260927 02:42:39.950243 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 18185
I20260927 02:42:39.950902 18459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:39.950358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44307 (local address 127.0.0.1:40292)
0927 02:42:39.950752 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:39.950819 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321}
W20260927 02:42:39.951701 17980 catalog_manager.cc:4584] TS c3d26992c5194753b86d7cd7391b9d3c (127.0.98.1:44307): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 21d8189c4a074a38826cd19bc5b99bbd: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44307: connect: Connection refused (error 111)
I20260927 02:42:39.955233 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 18316
I20260927 02:42:39.959682 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 17963
2026-09-27T02:42:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1352 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:42:40 dist-test-slave-zv8z krb5kdc[18490](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:42:40 dist-test-slave-zv8z krb5kdc[18490](info): set up 2 sockets
Sep 27 02:42:40 dist-test-slave-zv8z krb5kdc[18490](info): commencing operation
krb5kdc: starting...
W20260927 02:42:40.913841 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/krb5kdc/test-user.keytab.
Sep 27 02:42:40 dist-test-slave-zv8z krb5kdc[18490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:42:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:42:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:42:41.027195 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33661
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33661
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:41.131604 18503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:41.131856 18503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:41.131893 18503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:41.135790 18503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:42:41.135845 18503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:41.135864 18503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:41.135879 18503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:42:41.135892 18503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:42:41.141037 18503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33661
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:42:41.141995 18503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:41.143008 18503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:41.153254 18508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:41.153265 18509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:41.153265 18511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:41.153290 18503 server_base.cc:1034] running on GCE node
I20260927 02:42:41.154034 18503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:41.155159 18503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:41.156322 18503 hybrid_clock.cc:648] HybridClock initialized: now 1790476961156304 us; error 30 us; skew 500 ppm
Sep 27 02:42:41 dist-test-slave-zv8z krb5kdc[18490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:41.158921 18503 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:42:41.159721 18503 webserver.cc:460] Webserver started at http://127.0.98.62:44973/ using document root <none> and password file <none>
I20260927 02:42:41.160216 18503 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:41.160270 18503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:41.160431 18503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:41.162406 18503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/master-0/data/instance:
uuid: "8ef49a23ee054d36bdeb457e9e2deb61"
format_stamp: "Formatted at 2026-09-27 02:42:41 on dist-test-slave-zv8z"
server_key: "0629076bef53996438d8535eccd27af3"
server_key_iv: "f47ed5f50b42b0992d362367fa03c69d"
server_key_version: "encryptionkey@0"
I20260927 02:42:41.162896 18503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/master-0/wal/instance:
uuid: "8ef49a23ee054d36bdeb457e9e2deb61"
format_stamp: "Formatted at 2026-09-27 02:42:41 on dist-test-slave-zv8z"
server_key: "0629076bef53996438d8535eccd27af3"
server_key_iv: "f47ed5f50b42b0992d362367fa03c69d"
server_key_version: "encryptionkey@0"
I20260927 02:42:41.166353 18503 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:42:41.168569 18518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:41.169761 18503 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:42:41.169857 18503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "8ef49a23ee054d36bdeb457e9e2deb61"
format_stamp: "Formatted at 2026-09-27 02:42:41 on dist-test-slave-zv8z"
server_key: "0629076bef53996438d8535eccd27af3"
server_key_iv: "f47ed5f50b42b0992d362367fa03c69d"
server_key_version: "encryptionkey@0"
I20260927 02:42:41.169929 18503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:41.188874 18503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:41.192106 18503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:41.192305 18503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:41.199064 18503 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33661
I20260927 02:42:41.199085 18570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33661 every 8 connection(s)
I20260927 02:42:41.199992 18503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:42:41.202777 18571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:41.203109 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 18503
I20260927 02:42:41.203243 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/master-0/wal/instance
I20260927 02:42:41.203504 392 external_mini_cluster.cc:1428] Setting key 2c032d41c579b34e12f27974e6f850d9
I20260927 02:42:41.209327 18571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ef49a23ee054d36bdeb457e9e2deb61: Bootstrap starting.
I20260927 02:42:41.211879 18571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ef49a23ee054d36bdeb457e9e2deb61: Neither blocks nor log segments found. Creating new log.
Sep 27 02:42:41 dist-test-slave-zv8z krb5kdc[18490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:42:41.212571 18571 log.cc:824] T 00000000000000000000000000000000 P 8ef49a23ee054d36bdeb457e9e2deb61: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:41.214309 18571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ef49a23ee054d36bdeb457e9e2deb61: No bootstrap required, opened a new log
I20260927 02:42:41.216892 18574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:41.204855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34962 (local address 127.0.98.62:33661)
0927 02:42:41.205257 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:41.205271 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:41.205316 (+ 45us) server_negotiation.cc:407] Connection header received
0927 02:42:41.205949 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:41.205966 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:41.206319 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:41.206601 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:41.207494 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:41.208468 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:41.209608 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:41.210070 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:41.210181 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:41.212785 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:41.212790 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:41.212802 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:41.212829 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:41.214346 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:41.214792 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:41.214796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:41.214801 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:41.214905 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:41.215156 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:41.215158 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:41.215159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:41.215528 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:41.215661 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:41.215927 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 02:42:41.216182 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":122,"threads_started":1}
I20260927 02:42:41.218014 18571 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ef49a23ee054d36bdeb457e9e2deb61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ef49a23ee054d36bdeb457e9e2deb61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33661 } }
I20260927 02:42:41.218225 18571 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ef49a23ee054d36bdeb457e9e2deb61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:41.218299 18571 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ef49a23ee054d36bdeb457e9e2deb61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ef49a23ee054d36bdeb457e9e2deb61, State: Initialized, Role: FOLLOWER
I20260927 02:42:41.218683 18571 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ef49a23ee054d36bdeb457e9e2deb61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ef49a23ee054d36bdeb457e9e2deb61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33661 } }
I20260927 02:42:41.218812 18571 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ef49a23ee054d36bdeb457e9e2deb61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:42:41.218874 18571 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ef49a23ee054d36bdeb457e9e2deb61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:42:41.218963 18571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ef49a23ee054d36bdeb457e9e2deb61 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:41.219771 18571 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ef49a23ee054d36bdeb457e9e2deb61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ef49a23ee054d36bdeb457e9e2deb61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33661 } }
I20260927 02:42:41.220072 18571 leader_election.cc:302] T 00000000000000000000000000000000 P 8ef49a23ee054d36bdeb457e9e2deb61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ef49a23ee054d36bdeb457e9e2deb61; no voters:
I20260927 02:42:41.220331 18571 leader_election.cc:288] T 00000000000000000000000000000000 P 8ef49a23ee054d36bdeb457e9e2deb61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:42:41.220440 18576 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ef49a23ee054d36bdeb457e9e2deb61 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:41.220654 18576 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ef49a23ee054d36bdeb457e9e2deb61 [term 1 LEADER]: Becoming Leader. State: Replica: 8ef49a23ee054d36bdeb457e9e2deb61, State: Running, Role: LEADER
I20260927 02:42:41.220959 18576 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ef49a23ee054d36bdeb457e9e2deb61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ef49a23ee054d36bdeb457e9e2deb61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33661 } }
I20260927 02:42:41.221429 18571 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ef49a23ee054d36bdeb457e9e2deb61 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:42:41.222635 18577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ef49a23ee054d36bdeb457e9e2deb61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ef49a23ee054d36bdeb457e9e2deb61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ef49a23ee054d36bdeb457e9e2deb61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33661 } } }
I20260927 02:42:41.222875 18577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ef49a23ee054d36bdeb457e9e2deb61 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:41.222635 18578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ef49a23ee054d36bdeb457e9e2deb61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ef49a23ee054d36bdeb457e9e2deb61. Latest consensus state: current_term: 1 leader_uuid: "8ef49a23ee054d36bdeb457e9e2deb61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ef49a23ee054d36bdeb457e9e2deb61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33661 } } }
I20260927 02:42:41.222986 18578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ef49a23ee054d36bdeb457e9e2deb61 [sys.catalog]: This master's current role is: LEADER
W20260927 02:42:41.226085 18591 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8ef49a23ee054d36bdeb457e9e2deb61: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:42:41.226150 18591 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:42:41.226228 18588 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:42:41.226924 18588 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:42:41.231789 18588 catalog_manager.cc:1347] Generated new cluster ID: 51966ec94230485a8034f733ba6dbb24
I20260927 02:42:41.231851 18588 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:42:41.244298 18588 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:42:41.245502 18588 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:42:41.255724 18588 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ef49a23ee054d36bdeb457e9e2deb61: Generated new TSK 0
I20260927 02:42:41.256536 18588 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:42:41.281062 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33661
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:41.389385 18597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:41.389618 18597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:41.389659 18597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:41.393620 18597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:41.393673 18597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:41.393746 18597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:42:41.398958 18597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:42:41.399888 18597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:41.400833 18597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:41.409579 18603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:41.409586 18602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:41.409828 18597 server_base.cc:1034] running on GCE node
W20260927 02:42:41.409590 18605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:41.410310 18597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:41.410903 18597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:41.412055 18597 hybrid_clock.cc:648] HybridClock initialized: now 1790476961412030 us; error 35 us; skew 500 ppm
Sep 27 02:42:41 dist-test-slave-zv8z krb5kdc[18490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:41.414557 18597 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:42:41.415349 18597 webserver.cc:460] Webserver started at http://127.0.98.1:35287/ using document root <none> and password file <none>
I20260927 02:42:41.415858 18597 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:41.415921 18597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:41.416095 18597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:41.418007 18597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/ts-0/data/instance:
uuid: "89373d18ac274d97be0f44f527a02864"
format_stamp: "Formatted at 2026-09-27 02:42:41 on dist-test-slave-zv8z"
server_key: "8b12b836dbf2050a65dc7239f027b8bb"
server_key_iv: "4c674e70daaaa48cc189bd63025d5faf"
server_key_version: "encryptionkey@0"
I20260927 02:42:41.418535 18597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/ts-0/wal/instance:
uuid: "89373d18ac274d97be0f44f527a02864"
format_stamp: "Formatted at 2026-09-27 02:42:41 on dist-test-slave-zv8z"
server_key: "8b12b836dbf2050a65dc7239f027b8bb"
server_key_iv: "4c674e70daaaa48cc189bd63025d5faf"
server_key_version: "encryptionkey@0"
I20260927 02:42:41.421945 18597 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 02:42:41.424355 18612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:41.425482 18597 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:42:41.425580 18597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/ts-0/wal
uuid: "89373d18ac274d97be0f44f527a02864"
format_stamp: "Formatted at 2026-09-27 02:42:41 on dist-test-slave-zv8z"
server_key: "8b12b836dbf2050a65dc7239f027b8bb"
server_key_iv: "4c674e70daaaa48cc189bd63025d5faf"
server_key_version: "encryptionkey@0"
I20260927 02:42:41.425664 18597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:41.450439 18597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:41.453413 18597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:41.453581 18597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:41.454130 18597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:41.454945 18597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:41.454993 18597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:41.455044 18597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:41.455067 18597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:41.463311 18597 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37785
I20260927 02:42:41.463393 18725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37785 every 8 connection(s)
I20260927 02:42:41.464136 18597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:42:41.466706 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 18597
I20260927 02:42:41.466835 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:42:41.467140 392 external_mini_cluster.cc:1428] Setting key a138921cf1d82f204ff65813da0d9291
Sep 27 02:42:41 dist-test-slave-zv8z krb5kdc[18490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:42:41.477406 18574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:41.465811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35787 (local address 127.0.98.62:33661)
0927 02:42:41.465944 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:41.465947 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:41.466850 (+ 903us) server_negotiation.cc:407] Connection header received
0927 02:42:41.467999 (+ 1149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:41.468002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:41.468045 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:41.468126 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:41.469600 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:41.470147 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:41.471295 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:41.471523 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:41.471609 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:41.474145 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:41.474148 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:41.474150 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:41.474174 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:41.475486 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:41.476025 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:41.476027 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:41.476029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:41.476094 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:41.476428 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:41.476430 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:41.476432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:41.476599 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:41.476673 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:41.477153 (+ 480us) server_negotiation.cc:299] Negotiation successful
0927 02:42:41.477265 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 02:42:41.478205 18728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:41.466153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33661 (local address 127.0.98.1:35787)
0927 02:42:41.466687 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:41.466744 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:41.467795 (+ 1051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:41.468316 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:41.468325 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:41.468810 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:41.469441 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:41.469459 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:41.470295 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:41.470299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:41.471184 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:41.471193 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:41.471652 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:41.471656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:41.471809 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:41.472316 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:41.472332 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:41.473990 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:41.475597 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:41.475603 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:41.475618 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:41.475936 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:41.476168 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:41.476171 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:41.476173 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:41.476346 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:41.476712 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:41.476717 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:41.476954 (+ 237us) client_negotiation.cc:769] Sending connection context
0927 02:42:41.477115 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 02:42:41.477389 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":108,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 02:42:41.479423 18726 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33661
I20260927 02:42:41.479642 18726 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:41.480090 18726 heartbeater.cc:511] Master 127.0.98.62:33661 requested a full tablet report, sending...
I20260927 02:42:41.481590 18535 ts_manager.cc:194] Registered new tserver with Master: 89373d18ac274d97be0f44f527a02864 (127.0.98.1:37785)
I20260927 02:42:41.483129 18535 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35787
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:42:41.493932 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33661
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:41.600850 18731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:41.601104 18731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:41.601145 18731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:41.605078 18731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:41.605129 18731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:41.605187 18731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:42:41.610208 18731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:42:41.611193 18731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:41.612112 18731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:41.621349 18736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:41.621410 18731 server_base.cc:1034] running on GCE node
W20260927 02:42:41.621323 18737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:41.621338 18739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:41.621948 18731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:41.622581 18731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:41.623739 18731 hybrid_clock.cc:648] HybridClock initialized: now 1790476961623710 us; error 46 us; skew 500 ppm
Sep 27 02:42:41 dist-test-slave-zv8z krb5kdc[18490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:41.626202 18731 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:42:41.627045 18731 webserver.cc:460] Webserver started at http://127.0.98.2:40185/ using document root <none> and password file <none>
I20260927 02:42:41.627555 18731 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:41.627607 18731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:41.627769 18731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:41.629834 18731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/ts-1/data/instance:
uuid: "7193b3085f5a40e5bb437da697e9d3df"
format_stamp: "Formatted at 2026-09-27 02:42:41 on dist-test-slave-zv8z"
server_key: "853ed68ee83a2a031dce957e954d088c"
server_key_iv: "b7863994e84be0f49536d79d6c7ad460"
server_key_version: "encryptionkey@0"
I20260927 02:42:41.630370 18731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7193b3085f5a40e5bb437da697e9d3df"
format_stamp: "Formatted at 2026-09-27 02:42:41 on dist-test-slave-zv8z"
server_key: "853ed68ee83a2a031dce957e954d088c"
server_key_iv: "b7863994e84be0f49536d79d6c7ad460"
server_key_version: "encryptionkey@0"
I20260927 02:42:41.633790 18731 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:42:41.636121 18746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:41.637259 18731 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:42:41.637351 18731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/ts-1/wal
uuid: "7193b3085f5a40e5bb437da697e9d3df"
format_stamp: "Formatted at 2026-09-27 02:42:41 on dist-test-slave-zv8z"
server_key: "853ed68ee83a2a031dce957e954d088c"
server_key_iv: "b7863994e84be0f49536d79d6c7ad460"
server_key_version: "encryptionkey@0"
I20260927 02:42:41.637449 18731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:41.652855 18731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:41.656648 18731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:41.656838 18731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:41.657518 18731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:41.658514 18731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:41.658601 18731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:41.658684 18731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:41.658716 18731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:41.667510 18731 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42951
I20260927 02:42:41.667536 18859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42951 every 8 connection(s)
I20260927 02:42:41.668367 18731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:42:41.669343 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 18731
I20260927 02:42:41.669456 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:42:41.669765 392 external_mini_cluster.cc:1428] Setting key af14fca4c210002937e4bf54bf6722a6
Sep 27 02:42:41 dist-test-slave-zv8z krb5kdc[18490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 02:42:41.681684 18574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:41.670092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53773 (local address 127.0.98.62:33661)
0927 02:42:41.670219 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:41.670223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:41.671313 (+ 1090us) server_negotiation.cc:407] Connection header received
0927 02:42:41.672179 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:41.672182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:41.672262 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:41.672334 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:41.673554 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:41.674105 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:41.675245 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:41.675475 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:41.675559 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:41.678149 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:41.678152 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:41.678154 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:41.678176 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:41.679625 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:41.680117 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:41.680119 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:41.680121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:41.680198 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:41.680565 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:41.680568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:41.680569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:41.680799 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:41.680896 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:41.681446 (+ 550us) server_negotiation.cc:299] Negotiation successful
0927 02:42:41.681544 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 02:42:41.682765 18862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:41.670601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33661 (local address 127.0.98.2:53773)
0927 02:42:41.671182 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:41.671225 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:41.671983 (+ 758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:41.672520 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:41.672527 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:41.672888 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:41.673428 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:41.673444 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:41.674206 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:41.674209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:41.675129 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:41.675145 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:41.675601 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:41.675605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:41.675771 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:41.676309 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:41.676325 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:41.677963 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:41.679782 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:41.679787 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:41.679798 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:41.680038 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:41.680319 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:41.680322 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:41.680323 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:41.680460 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:41.680932 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:41.680937 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:41.681193 (+ 256us) client_negotiation.cc:769] Sending connection context
0927 02:42:41.681386 (+ 193us) client_negotiation.cc:240] Negotiation successful
0927 02:42:41.681725 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":137,"threads_started":1}
I20260927 02:42:41.683974 18860 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33661
I20260927 02:42:41.684212 18860 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:41.684760 18860 heartbeater.cc:511] Master 127.0.98.62:33661 requested a full tablet report, sending...
I20260927 02:42:41.685777 18535 ts_manager.cc:194] Registered new tserver with Master: 7193b3085f5a40e5bb437da697e9d3df (127.0.98.2:42951)
I20260927 02:42:41.686354 18535 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53773
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:42:41.696724 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33661
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:41.801795 18865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:41.802032 18865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:41.802073 18865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:41.805994 18865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:41.806046 18865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:41.806098 18865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:42:41.811182 18865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:42:41.812098 18865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:41.813000 18865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:42:41.821916 18865 server_base.cc:1034] running on GCE node
W20260927 02:42:41.821882 18871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:41.821899 18870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:41.821882 18873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:41.822578 18865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:41.823266 18865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:41.824425 18865 hybrid_clock.cc:648] HybridClock initialized: now 1790476961824404 us; error 33 us; skew 500 ppm
Sep 27 02:42:41 dist-test-slave-zv8z krb5kdc[18490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:41.827010 18865 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:42:41.827870 18865 webserver.cc:460] Webserver started at http://127.0.98.3:35959/ using document root <none> and password file <none>
I20260927 02:42:41.828382 18865 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:41.828440 18865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:41.828647 18865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:41.830639 18865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/ts-2/data/instance:
uuid: "7409ee0901044ea383cd7f874aa6c5f3"
format_stamp: "Formatted at 2026-09-27 02:42:41 on dist-test-slave-zv8z"
server_key: "dabdc3a7fc4c5653883c8e3b3608c71f"
server_key_iv: "2502300da486e6f5b5ca4e40687749ab"
server_key_version: "encryptionkey@0"
I20260927 02:42:41.831173 18865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7409ee0901044ea383cd7f874aa6c5f3"
format_stamp: "Formatted at 2026-09-27 02:42:41 on dist-test-slave-zv8z"
server_key: "dabdc3a7fc4c5653883c8e3b3608c71f"
server_key_iv: "2502300da486e6f5b5ca4e40687749ab"
server_key_version: "encryptionkey@0"
I20260927 02:42:41.834605 18865 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:42:41.836858 18880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:41.838001 18865 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:42:41.838099 18865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/ts-2/wal
uuid: "7409ee0901044ea383cd7f874aa6c5f3"
format_stamp: "Formatted at 2026-09-27 02:42:41 on dist-test-slave-zv8z"
server_key: "dabdc3a7fc4c5653883c8e3b3608c71f"
server_key_iv: "2502300da486e6f5b5ca4e40687749ab"
server_key_version: "encryptionkey@0"
I20260927 02:42:41.838186 18865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:41.872378 18865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:41.875528 18865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:41.875705 18865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:41.876281 18865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:41.877063 18865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:41.877108 18865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:41.877158 18865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:41.877187 18865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:41.885228 18865 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42527
I20260927 02:42:41.885249 18993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42527 every 8 connection(s)
I20260927 02:42:41.886049 18865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:42:41.892619 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 18865
I20260927 02:42:41.892720 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:42:41.892972 392 external_mini_cluster.cc:1428] Setting key f097e98dd6667c79a216a4111c22ed35
Sep 27 02:42:41 dist-test-slave-zv8z krb5kdc[18490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:42:41.898523 18574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:41.887722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44743 (local address 127.0.98.62:33661)
0927 02:42:41.887877 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:41.887882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:41.888702 (+ 820us) server_negotiation.cc:407] Connection header received
0927 02:42:41.889565 (+ 863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:41.889568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:41.889615 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:41.889687 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:41.890844 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:41.891416 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:41.892616 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:41.892904 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:41.892994 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:41.895653 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:41.895656 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:41.895658 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:41.895681 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:41.896863 (+ 1182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:41.897296 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:41.897298 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:41.897300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:41.897372 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:41.897636 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:41.897638 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:41.897640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:41.897805 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:41.897876 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:41.898313 (+ 437us) server_negotiation.cc:299] Negotiation successful
0927 02:42:41.898410 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 02:42:41.899303 18996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:41.888106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33661 (local address 127.0.98.3:44743)
0927 02:42:41.888551 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:41.888592 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:41.889375 (+ 783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:41.889842 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:41.889849 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:41.890184 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:41.890690 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:41.890702 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:41.891550 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:41.891552 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:41.892480 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:41.892491 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:41.893056 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:41.893060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:41.893252 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:41.893848 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:41.893869 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:41.895509 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:41.896967 (+ 1458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:41.896971 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:41.896983 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:41.897218 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:41.897438 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:41.897439 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:41.897441 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:41.897569 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:41.897881 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:41.897886 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:41.898099 (+ 213us) client_negotiation.cc:769] Sending connection context
0927 02:42:41.898269 (+ 170us) client_negotiation.cc:240] Negotiation successful
0927 02:42:41.898542 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":103,"threads_started":1}
I20260927 02:42:41.900362 18994 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33661
I20260927 02:42:41.900597 18994 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:41.901153 18994 heartbeater.cc:511] Master 127.0.98.62:33661 requested a full tablet report, sending...
I20260927 02:42:41.902222 18535 ts_manager.cc:194] Registered new tserver with Master: 7409ee0901044ea383cd7f874aa6c5f3 (127.0.98.3:42527)
I20260927 02:42:41.902819 18535 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44743
I20260927 02:42:41.906838 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:42:41.915565 18574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:41.908595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34974 (local address 127.0.98.62:33661)
0927 02:42:41.908701 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:41.908706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:41.908901 (+ 195us) server_negotiation.cc:407] Connection header received
0927 02:42:41.909003 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:41.909005 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:41.909051 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:41.909112 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:41.909785 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:41.910319 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:41.911089 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:41.911295 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:41.911376 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:41.912749 (+ 1373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:41.912753 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:41.912755 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:41.912776 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:41.913883 (+ 1107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:41.914368 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:41.914372 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:41.914374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:41.914446 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:41.914753 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:41.914757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:41.914758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:41.914943 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:41.915082 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:41.915337 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 02:42:41.915417 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 02:42:41.918130 18535 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34974:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:42:41.921067 18535 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:42:41.934715 19008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:41.929481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42951 (local address 127.0.0.1:36726)
0927 02:42:41.929745 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:41.929758 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:41.929841 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:41.930542 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:41.930545 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:41.930561 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:41.930645 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:41.930651 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:41.932008 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:41.932012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:41.933077 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:41.933086 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:41.934125 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:41.934129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:41.934208 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:41.934236 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 02:42:41.934376 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 02:42:41.934502 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":57,"thread_start_us":80,"threads_started":1}
I20260927 02:42:41.934846 19006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:41.928857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60564 (local address 127.0.98.1:37785)
0927 02:42:41.929472 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:41.929485 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:41.929509 (+ 24us) server_negotiation.cc:407] Connection header received
0927 02:42:41.929581 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:41.929586 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:41.929788 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:41.930018 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:41.930586 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:41.931744 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:41.933502 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:41.934289 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:41.934387 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:41.934519 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:41.934599 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 02:42:41.934712 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":77,"threads_started":1}
I20260927 02:42:41.935340 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:41.929517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36726 (local address 127.0.98.2:42951)
0927 02:42:41.929910 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:41.929920 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:41.929940 (+ 20us) server_negotiation.cc:407] Connection header received
0927 02:42:41.930009 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:41.930013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:41.930205 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:41.930380 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:41.930765 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:41.931889 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:41.933222 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:41.933994 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:41.934106 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:41.934291 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:41.935097 (+ 806us) server_negotiation.cc:299] Negotiation successful
0927 02:42:41.935195 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":92,"threads_started":1}
I20260927 02:42:41.935635 19007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:41.928932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42527 (local address 127.0.0.1:36018)
0927 02:42:41.929195 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:41.929207 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:41.929275 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:41.930829 (+ 1554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:41.930832 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:41.930848 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:41.930939 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:41.930945 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:41.931868 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:41.931871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:41.933379 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:41.933390 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:41.935304 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:41.935307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:41.935382 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:41.935406 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 02:42:41.935480 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 02:42:41.935540 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":120,"thread_start_us":82,"threads_started":1}
I20260927 02:42:41.935734 19009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:41.929484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36018 (local address 127.0.98.3:42527)
0927 02:42:41.930411 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:41.930425 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:41.930445 (+ 20us) server_negotiation.cc:407] Connection header received
0927 02:42:41.930511 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:41.930515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:41.930679 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:41.930840 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:41.931035 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:41.931752 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:41.934363 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:41.935188 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:41.935293 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:41.935425 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:41.935506 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 02:42:41.935593 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":835,"thread_start_us":120,"threads_started":1}
I20260927 02:42:41.934836 19005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:41.928681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37785 (local address 127.0.0.1:60564)
0927 02:42:41.929040 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:41.929087 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:41.929203 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:41.930026 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:41.930032 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:41.930056 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:41.930219 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:41.930226 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:41.931890 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:41.931892 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:41.933379 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:41.933390 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:41.934388 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:41.934392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:41.934477 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:41.934498 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 02:42:41.934567 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 02:42:41.934653 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":96,"thread_start_us":67,"threads_started":1}
I20260927 02:42:41.938210 18660 tablet_service.cc:1467] Processing CreateTablet for tablet 68ba3792529f4145a91c9748b15a5572 (DEFAULT_TABLE table=test-table [id=f95cf1b73c2f4dbfb16ab94883254b34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:41.938385 18928 tablet_service.cc:1467] Processing CreateTablet for tablet 68ba3792529f4145a91c9748b15a5572 (DEFAULT_TABLE table=test-table [id=f95cf1b73c2f4dbfb16ab94883254b34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:41.938385 18794 tablet_service.cc:1467] Processing CreateTablet for tablet 68ba3792529f4145a91c9748b15a5572 (DEFAULT_TABLE table=test-table [id=f95cf1b73c2f4dbfb16ab94883254b34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:41.939224 18660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68ba3792529f4145a91c9748b15a5572. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:41.939353 18928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68ba3792529f4145a91c9748b15a5572. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:41.939353 18794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68ba3792529f4145a91c9748b15a5572. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:41.946527 19012 tablet_bootstrap.cc:492] T 68ba3792529f4145a91c9748b15a5572 P 7409ee0901044ea383cd7f874aa6c5f3: Bootstrap starting.
I20260927 02:42:41.946527 19011 tablet_bootstrap.cc:492] T 68ba3792529f4145a91c9748b15a5572 P 89373d18ac274d97be0f44f527a02864: Bootstrap starting.
I20260927 02:42:41.948737 19011 tablet_bootstrap.cc:654] T 68ba3792529f4145a91c9748b15a5572 P 89373d18ac274d97be0f44f527a02864: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:41.948737 19012 tablet_bootstrap.cc:654] T 68ba3792529f4145a91c9748b15a5572 P 7409ee0901044ea383cd7f874aa6c5f3: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:41.949119 19013 tablet_bootstrap.cc:492] T 68ba3792529f4145a91c9748b15a5572 P 7193b3085f5a40e5bb437da697e9d3df: Bootstrap starting.
I20260927 02:42:41.949924 19011 log.cc:824] T 68ba3792529f4145a91c9748b15a5572 P 89373d18ac274d97be0f44f527a02864: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:41.949924 19012 log.cc:824] T 68ba3792529f4145a91c9748b15a5572 P 7409ee0901044ea383cd7f874aa6c5f3: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:41.951323 19013 tablet_bootstrap.cc:654] T 68ba3792529f4145a91c9748b15a5572 P 7193b3085f5a40e5bb437da697e9d3df: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:41.951862 19011 tablet_bootstrap.cc:492] T 68ba3792529f4145a91c9748b15a5572 P 89373d18ac274d97be0f44f527a02864: No bootstrap required, opened a new log
I20260927 02:42:41.951867 19012 tablet_bootstrap.cc:492] T 68ba3792529f4145a91c9748b15a5572 P 7409ee0901044ea383cd7f874aa6c5f3: No bootstrap required, opened a new log
I20260927 02:42:41.951956 19013 log.cc:824] T 68ba3792529f4145a91c9748b15a5572 P 7193b3085f5a40e5bb437da697e9d3df: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:41.952049 19012 ts_tablet_manager.cc:1397] T 68ba3792529f4145a91c9748b15a5572 P 7409ee0901044ea383cd7f874aa6c5f3: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260927 02:42:41.952049 19011 ts_tablet_manager.cc:1397] T 68ba3792529f4145a91c9748b15a5572 P 89373d18ac274d97be0f44f527a02864: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20260927 02:42:41.953475 19013 tablet_bootstrap.cc:492] T 68ba3792529f4145a91c9748b15a5572 P 7193b3085f5a40e5bb437da697e9d3df: No bootstrap required, opened a new log
I20260927 02:42:41.953634 19013 ts_tablet_manager.cc:1397] T 68ba3792529f4145a91c9748b15a5572 P 7193b3085f5a40e5bb437da697e9d3df: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 02:42:41.955754 19011 raft_consensus.cc:348] T 68ba3792529f4145a91c9748b15a5572 P 89373d18ac274d97be0f44f527a02864 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89373d18ac274d97be0f44f527a02864" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37785 } } peers { permanent_uuid: "7409ee0901044ea383cd7f874aa6c5f3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42527 } } peers { permanent_uuid: "7193b3085f5a40e5bb437da697e9d3df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42951 } }
I20260927 02:42:41.955754 19012 raft_consensus.cc:348] T 68ba3792529f4145a91c9748b15a5572 P 7409ee0901044ea383cd7f874aa6c5f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89373d18ac274d97be0f44f527a02864" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37785 } } peers { permanent_uuid: "7409ee0901044ea383cd7f874aa6c5f3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42527 } } peers { permanent_uuid: "7193b3085f5a40e5bb437da697e9d3df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42951 } }
I20260927 02:42:41.956009 19012 raft_consensus.cc:374] T 68ba3792529f4145a91c9748b15a5572 P 7409ee0901044ea383cd7f874aa6c5f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:41.956009 19011 raft_consensus.cc:374] T 68ba3792529f4145a91c9748b15a5572 P 89373d18ac274d97be0f44f527a02864 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:41.956064 19011 raft_consensus.cc:729] T 68ba3792529f4145a91c9748b15a5572 P 89373d18ac274d97be0f44f527a02864 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89373d18ac274d97be0f44f527a02864, State: Initialized, Role: FOLLOWER
I20260927 02:42:41.956089 19012 raft_consensus.cc:729] T 68ba3792529f4145a91c9748b15a5572 P 7409ee0901044ea383cd7f874aa6c5f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7409ee0901044ea383cd7f874aa6c5f3, State: Initialized, Role: FOLLOWER
I20260927 02:42:41.956267 19013 raft_consensus.cc:348] T 68ba3792529f4145a91c9748b15a5572 P 7193b3085f5a40e5bb437da697e9d3df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89373d18ac274d97be0f44f527a02864" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37785 } } peers { permanent_uuid: "7409ee0901044ea383cd7f874aa6c5f3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42527 } } peers { permanent_uuid: "7193b3085f5a40e5bb437da697e9d3df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42951 } }
I20260927 02:42:41.956501 19013 raft_consensus.cc:374] T 68ba3792529f4145a91c9748b15a5572 P 7193b3085f5a40e5bb437da697e9d3df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:41.956570 19013 raft_consensus.cc:729] T 68ba3792529f4145a91c9748b15a5572 P 7193b3085f5a40e5bb437da697e9d3df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7193b3085f5a40e5bb437da697e9d3df, State: Initialized, Role: FOLLOWER
I20260927 02:42:41.956533 19012 consensus_queue.cc:260] T 68ba3792529f4145a91c9748b15a5572 P 7409ee0901044ea383cd7f874aa6c5f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89373d18ac274d97be0f44f527a02864" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37785 } } peers { permanent_uuid: "7409ee0901044ea383cd7f874aa6c5f3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42527 } } peers { permanent_uuid: "7193b3085f5a40e5bb437da697e9d3df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42951 } }
I20260927 02:42:41.956533 19011 consensus_queue.cc:260] T 68ba3792529f4145a91c9748b15a5572 P 89373d18ac274d97be0f44f527a02864 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89373d18ac274d97be0f44f527a02864" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37785 } } peers { permanent_uuid: "7409ee0901044ea383cd7f874aa6c5f3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42527 } } peers { permanent_uuid: "7193b3085f5a40e5bb437da697e9d3df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42951 } }
I20260927 02:42:41.956971 19013 consensus_queue.cc:260] T 68ba3792529f4145a91c9748b15a5572 P 7193b3085f5a40e5bb437da697e9d3df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89373d18ac274d97be0f44f527a02864" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37785 } } peers { permanent_uuid: "7409ee0901044ea383cd7f874aa6c5f3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42527 } } peers { permanent_uuid: "7193b3085f5a40e5bb437da697e9d3df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42951 } }
I20260927 02:42:41.957481 19012 ts_tablet_manager.cc:1428] T 68ba3792529f4145a91c9748b15a5572 P 7409ee0901044ea383cd7f874aa6c5f3: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260927 02:42:41.957481 19011 ts_tablet_manager.cc:1428] T 68ba3792529f4145a91c9748b15a5572 P 89373d18ac274d97be0f44f527a02864: Time spent starting tablet: real 0.005s user 0.002s sys 0.004s
I20260927 02:42:41.957823 18994 heartbeater.cc:503] Master 127.0.98.62:33661 was elected leader, sending a full tablet report...
I20260927 02:42:41.957883 19013 ts_tablet_manager.cc:1428] T 68ba3792529f4145a91c9748b15a5572 P 7193b3085f5a40e5bb437da697e9d3df: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260927 02:42:41.958187 18860 heartbeater.cc:503] Master 127.0.98.62:33661 was elected leader, sending a full tablet report...
I20260927 02:42:41.958498 18726 heartbeater.cc:503] Master 127.0.98.62:33661 was elected leader, sending a full tablet report...
W20260927 02:42:41.965130 18727 tablet.cc:2369] T 68ba3792529f4145a91c9748b15a5572 P 89373d18ac274d97be0f44f527a02864: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:42:42.009569 19019 raft_consensus.cc:482] T 68ba3792529f4145a91c9748b15a5572 P 7193b3085f5a40e5bb437da697e9d3df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:42:42.009840 19019 raft_consensus.cc:504] T 68ba3792529f4145a91c9748b15a5572 P 7193b3085f5a40e5bb437da697e9d3df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89373d18ac274d97be0f44f527a02864" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37785 } } peers { permanent_uuid: "7409ee0901044ea383cd7f874aa6c5f3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42527 } } peers { permanent_uuid: "7193b3085f5a40e5bb437da697e9d3df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42951 } }
I20260927 02:42:42.011030 19019 leader_election.cc:288] T 68ba3792529f4145a91c9748b15a5572 P 7193b3085f5a40e5bb437da697e9d3df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89373d18ac274d97be0f44f527a02864 (127.0.98.1:37785), 7409ee0901044ea383cd7f874aa6c5f3 (127.0.98.3:42527)
I20260927 02:42:42.014355 19006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:42.011318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44505 (local address 127.0.98.1:37785)
0927 02:42:42.011422 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:42.011427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:42.011551 (+ 124us) server_negotiation.cc:407] Connection header received
0927 02:42:42.011609 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:42.011612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:42.011653 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:42.011722 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:42.011959 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:42.012438 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:42.013439 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:42.013819 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:42.013941 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:42.014003 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:42.014151 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 02:42:42.014217 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 02:42:42.014355 18862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:42.011250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37785 (local address 127.0.98.2:44505)
0927 02:42:42.011420 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:42.011434 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:42.011525 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:42.011740 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:42.011743 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:42.011759 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:42.011859 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:42.011864 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:42.012535 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:42.012538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:42.013317 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:42.013325 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:42.013944 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:42.013947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:42.014046 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:42.014064 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 02:42:42.014127 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 02:42:42.014206 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260927 02:42:42.014523 19009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:42.011405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47593 (local address 127.0.98.3:42527)
0927 02:42:42.011523 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:42.011527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:42.011720 (+ 193us) server_negotiation.cc:407] Connection header received
0927 02:42:42.011844 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:42.011847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:42.011890 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:42.011950 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:42.012174 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:42.012651 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:42.013608 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:42.014010 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:42.014101 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:42.014159 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:42.014356 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 02:42:42.014420 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 02:42:42.014537 19020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:42.011280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42527 (local address 127.0.98.2:47593)
0927 02:42:42.011629 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:42.011641 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:42.011740 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:42.011967 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:42.011970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:42.011984 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:42.012082 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:42.012087 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:42.012746 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:42.012749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:42.013497 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:42.013504 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:42.014113 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:42.014117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:42.014220 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:42.014237 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 02:42:42.014332 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 02:42:42.014396 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":1,"thread_start_us":127,"threads_started":1}
I20260927 02:42:42.015091 18948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68ba3792529f4145a91c9748b15a5572" candidate_uuid: "7193b3085f5a40e5bb437da697e9d3df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7409ee0901044ea383cd7f874aa6c5f3" is_pre_election: true
I20260927 02:42:42.015100 18680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68ba3792529f4145a91c9748b15a5572" candidate_uuid: "7193b3085f5a40e5bb437da697e9d3df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89373d18ac274d97be0f44f527a02864" is_pre_election: true
I20260927 02:42:42.015352 18680 raft_consensus.cc:2454] T 68ba3792529f4145a91c9748b15a5572 P 89373d18ac274d97be0f44f527a02864 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7193b3085f5a40e5bb437da697e9d3df in term 0.
I20260927 02:42:42.015363 18948 raft_consensus.cc:2454] T 68ba3792529f4145a91c9748b15a5572 P 7409ee0901044ea383cd7f874aa6c5f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7193b3085f5a40e5bb437da697e9d3df in term 0.
I20260927 02:42:42.015858 18749 leader_election.cc:302] T 68ba3792529f4145a91c9748b15a5572 P 7193b3085f5a40e5bb437da697e9d3df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7193b3085f5a40e5bb437da697e9d3df, 7409ee0901044ea383cd7f874aa6c5f3; no voters:
I20260927 02:42:42.016191 19019 raft_consensus.cc:2781] T 68ba3792529f4145a91c9748b15a5572 P 7193b3085f5a40e5bb437da697e9d3df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:42:42.016279 19019 raft_consensus.cc:482] T 68ba3792529f4145a91c9748b15a5572 P 7193b3085f5a40e5bb437da697e9d3df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:42:42.016309 19019 raft_consensus.cc:3037] T 68ba3792529f4145a91c9748b15a5572 P 7193b3085f5a40e5bb437da697e9d3df [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:42.017284 19019 raft_consensus.cc:504] T 68ba3792529f4145a91c9748b15a5572 P 7193b3085f5a40e5bb437da697e9d3df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89373d18ac274d97be0f44f527a02864" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37785 } } peers { permanent_uuid: "7409ee0901044ea383cd7f874aa6c5f3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42527 } } peers { permanent_uuid: "7193b3085f5a40e5bb437da697e9d3df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42951 } }
I20260927 02:42:42.017607 19019 leader_election.cc:288] T 68ba3792529f4145a91c9748b15a5572 P 7193b3085f5a40e5bb437da697e9d3df [CANDIDATE]: Term 1 election: Requested vote from peers 89373d18ac274d97be0f44f527a02864 (127.0.98.1:37785), 7409ee0901044ea383cd7f874aa6c5f3 (127.0.98.3:42527)
I20260927 02:42:42.017958 18680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68ba3792529f4145a91c9748b15a5572" candidate_uuid: "7193b3085f5a40e5bb437da697e9d3df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89373d18ac274d97be0f44f527a02864"
I20260927 02:42:42.017949 18948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68ba3792529f4145a91c9748b15a5572" candidate_uuid: "7193b3085f5a40e5bb437da697e9d3df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7409ee0901044ea383cd7f874aa6c5f3"
I20260927 02:42:42.018085 18948 raft_consensus.cc:3037] T 68ba3792529f4145a91c9748b15a5572 P 7409ee0901044ea383cd7f874aa6c5f3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:42.018081 18680 raft_consensus.cc:3037] T 68ba3792529f4145a91c9748b15a5572 P 89373d18ac274d97be0f44f527a02864 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:42.019111 18948 raft_consensus.cc:2454] T 68ba3792529f4145a91c9748b15a5572 P 7409ee0901044ea383cd7f874aa6c5f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7193b3085f5a40e5bb437da697e9d3df in term 1.
I20260927 02:42:42.019165 18680 raft_consensus.cc:2454] T 68ba3792529f4145a91c9748b15a5572 P 89373d18ac274d97be0f44f527a02864 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7193b3085f5a40e5bb437da697e9d3df in term 1.
I20260927 02:42:42.019413 18749 leader_election.cc:302] T 68ba3792529f4145a91c9748b15a5572 P 7193b3085f5a40e5bb437da697e9d3df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7193b3085f5a40e5bb437da697e9d3df, 7409ee0901044ea383cd7f874aa6c5f3; no voters:
I20260927 02:42:42.019567 19019 raft_consensus.cc:2781] T 68ba3792529f4145a91c9748b15a5572 P 7193b3085f5a40e5bb437da697e9d3df [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:42.019773 19019 raft_consensus.cc:686] T 68ba3792529f4145a91c9748b15a5572 P 7193b3085f5a40e5bb437da697e9d3df [term 1 LEADER]: Becoming Leader. State: Replica: 7193b3085f5a40e5bb437da697e9d3df, State: Running, Role: LEADER
I20260927 02:42:42.020097 19019 consensus_queue.cc:237] T 68ba3792529f4145a91c9748b15a5572 P 7193b3085f5a40e5bb437da697e9d3df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89373d18ac274d97be0f44f527a02864" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37785 } } peers { permanent_uuid: "7409ee0901044ea383cd7f874aa6c5f3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42527 } } peers { permanent_uuid: "7193b3085f5a40e5bb437da697e9d3df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42951 } }
I20260927 02:42:42.023126 18535 catalog_manager.cc:5495] T 68ba3792529f4145a91c9748b15a5572 P 7193b3085f5a40e5bb437da697e9d3df reported cstate change: term changed from 0 to 1, leader changed from <none> to 7193b3085f5a40e5bb437da697e9d3df (127.0.98.2). New cstate: current_term: 1 leader_uuid: "7193b3085f5a40e5bb437da697e9d3df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89373d18ac274d97be0f44f527a02864" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7409ee0901044ea383cd7f874aa6c5f3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7193b3085f5a40e5bb437da697e9d3df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42951 } health_report { overall_health: HEALTHY } } }
I20260927 02:42:42.055881 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:42.052310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36736 (local address 127.0.98.2:42951)
0927 02:42:42.052453 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:42.052458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:42.052473 (+ 15us) server_negotiation.cc:407] Connection header received
0927 02:42:42.052574 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:42.052577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:42.052627 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:42.052714 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:42:42.053093 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:42.053587 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:42.054481 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:42.054700 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:42.054794 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:42.055069 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:42:42.055464 (+ 395us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:42:42.055591 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:42.055698 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 02:42:42.055735 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 02:42:42.061681 18948 raft_consensus.cc:1261] T 68ba3792529f4145a91c9748b15a5572 P 7409ee0901044ea383cd7f874aa6c5f3 [term 1 FOLLOWER]: Refusing update from remote peer 7193b3085f5a40e5bb437da697e9d3df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:42:42.061685 18680 raft_consensus.cc:1261] T 68ba3792529f4145a91c9748b15a5572 P 89373d18ac274d97be0f44f527a02864 [term 1 FOLLOWER]: Refusing update from remote peer 7193b3085f5a40e5bb437da697e9d3df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:42:42.062300 19019 consensus_queue.cc:1035] T 68ba3792529f4145a91c9748b15a5572 P 7193b3085f5a40e5bb437da697e9d3df [LEADER]: Connected to new peer: Peer: permanent_uuid: "89373d18ac274d97be0f44f527a02864" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:42:42.062484 19021 consensus_queue.cc:1035] T 68ba3792529f4145a91c9748b15a5572 P 7193b3085f5a40e5bb437da697e9d3df [LEADER]: Connected to new peer: Peer: permanent_uuid: "7409ee0901044ea383cd7f874aa6c5f3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:42:42.070046 19024 mvcc.cc:205] Tried to move back new op lower bound from 7333793636596994048 to 7333793636436787200. Current Snapshot: MvccSnapshot[applied={T|T < 7333793636596994048}]
I20260927 02:42:42.071120 19026 mvcc.cc:205] Tried to move back new op lower bound from 7333793636596994048 to 7333793636436787200. Current Snapshot: MvccSnapshot[applied={T|T < 7333793636596994048}]
I20260927 02:42:42.073424 19025 mvcc.cc:205] Tried to move back new op lower bound from 7333793636596994048 to 7333793636436787200. Current Snapshot: MvccSnapshot[applied={T|T < 7333793636596994048 or (T in {7333793636596994048})}]
I20260927 02:42:42.077718 18535 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:42:42.077848 18535 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:42:42.080106 18535 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ef49a23ee054d36bdeb457e9e2deb61: Sending DeleteTablet for 3 replicas of tablet 68ba3792529f4145a91c9748b15a5572
I20260927 02:42:42.080837 18928 tablet_service.cc:1514] Processing DeleteTablet for tablet 68ba3792529f4145a91c9748b15a5572 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36018
I20260927 02:42:42.080870 18660 tablet_service.cc:1514] Processing DeleteTablet for tablet 68ba3792529f4145a91c9748b15a5572 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60564
I20260927 02:42:42.080937 18794 tablet_service.cc:1514] Processing DeleteTablet for tablet 68ba3792529f4145a91c9748b15a5572 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36726
I20260927 02:42:42.081310 19035 tablet_replica.cc:330] T 68ba3792529f4145a91c9748b15a5572 P 7409ee0901044ea383cd7f874aa6c5f3: stopping tablet replica
I20260927 02:42:42.081446 19033 tablet_replica.cc:330] T 68ba3792529f4145a91c9748b15a5572 P 89373d18ac274d97be0f44f527a02864: stopping tablet replica
I20260927 02:42:42.081642 19035 raft_consensus.cc:2229] T 68ba3792529f4145a91c9748b15a5572 P 7409ee0901044ea383cd7f874aa6c5f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:42:42.081696 19033 raft_consensus.cc:2229] T 68ba3792529f4145a91c9748b15a5572 P 89373d18ac274d97be0f44f527a02864 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:42:42.081938 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 18597
I20260927 02:42:42.081986 19033 raft_consensus.cc:2258] T 68ba3792529f4145a91c9748b15a5572 P 89373d18ac274d97be0f44f527a02864 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:42.081986 19035 raft_consensus.cc:2258] T 68ba3792529f4145a91c9748b15a5572 P 7409ee0901044ea383cd7f874aa6c5f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:42.083123 19034 tablet_replica.cc:330] T 68ba3792529f4145a91c9748b15a5572 P 7193b3085f5a40e5bb437da697e9d3df: stopping tablet replica
I20260927 02:42:42.083366 19034 raft_consensus.cc:2229] T 68ba3792529f4145a91c9748b15a5572 P 7193b3085f5a40e5bb437da697e9d3df [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:42:42.083699 19035 ts_tablet_manager.cc:1905] T 68ba3792529f4145a91c9748b15a5572 P 7409ee0901044ea383cd7f874aa6c5f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:42:42.083700 19034 raft_consensus.cc:2258] T 68ba3792529f4145a91c9748b15a5572 P 7193b3085f5a40e5bb437da697e9d3df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:42.084967 19034 ts_tablet_manager.cc:1905] T 68ba3792529f4145a91c9748b15a5572 P 7193b3085f5a40e5bb437da697e9d3df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:42:42.086181 19035 ts_tablet_manager.cc:1918] T 68ba3792529f4145a91c9748b15a5572 P 7409ee0901044ea383cd7f874aa6c5f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:42:42.086285 19035 log.cc:1196] T 68ba3792529f4145a91c9748b15a5572 P 7409ee0901044ea383cd7f874aa6c5f3: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/ts-2/wal/wals/68ba3792529f4145a91c9748b15a5572
I20260927 02:42:42.086632 19035 ts_tablet_manager.cc:1939] T 68ba3792529f4145a91c9748b15a5572 P 7409ee0901044ea383cd7f874aa6c5f3: Deleting consensus metadata
I20260927 02:42:42.087414 19034 ts_tablet_manager.cc:1918] T 68ba3792529f4145a91c9748b15a5572 P 7193b3085f5a40e5bb437da697e9d3df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:42:42.087550 19034 log.cc:1196] T 68ba3792529f4145a91c9748b15a5572 P 7193b3085f5a40e5bb437da697e9d3df: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790476718023131-392-0/minicluster-data/ts-1/wal/wals/68ba3792529f4145a91c9748b15a5572
I20260927 02:42:42.087834 18521 catalog_manager.cc:4841] TS 7409ee0901044ea383cd7f874aa6c5f3 (127.0.98.3:42527): tablet 68ba3792529f4145a91c9748b15a5572 (table test-table [id=f95cf1b73c2f4dbfb16ab94883254b34]) successfully deleted
I20260927 02:42:42.087900 19034 ts_tablet_manager.cc:1939] T 68ba3792529f4145a91c9748b15a5572 P 7193b3085f5a40e5bb437da697e9d3df: Deleting consensus metadata
I20260927 02:42:42.088716 18520 catalog_manager.cc:4841] TS 7193b3085f5a40e5bb437da697e9d3df (127.0.98.2:42951): tablet 68ba3792529f4145a91c9748b15a5572 (table test-table [id=f95cf1b73c2f4dbfb16ab94883254b34]) successfully deleted
W20260927 02:42:42.091372 18520 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37785 (error 108)
I20260927 02:42:42.091405 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 18731
I20260927 02:42:42.092555 19005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:42.092295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37785 (local address 127.0.0.1:60566)
0927 02:42:42.092410 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:42.092474 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 02:42:42.092799 18520 catalog_manager.cc:4584] TS 89373d18ac274d97be0f44f527a02864 (127.0.98.1:37785): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 68ba3792529f4145a91c9748b15a5572: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37785: connect: Connection refused (error 111)
I20260927 02:42:42.096537 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 18865
I20260927 02:42:42.100873 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 18503
2026-09-27T02:42:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2141 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T02:42:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:42:42Z Disabled control of system clock
I20260927 02:42:42.128017 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36539
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 02:42:42.233001 19040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:42.233255 19040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:42.233289 19040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:42.237239 19040 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:42:42.237298 19040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:42.237311 19040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:42.237327 19040 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:42:42.237340 19040 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:42:42.241950 19040 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36539
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:42:42.242961 19040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:42.244211 19040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:42.252105 19048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:42.252120 19045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:42.252215 19040 server_base.cc:1034] running on GCE node
W20260927 02:42:42.252285 19046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:42.252910 19040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:42.253998 19040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:42.255174 19040 hybrid_clock.cc:648] HybridClock initialized: now 1790476962255137 us; error 46 us; skew 500 ppm
I20260927 02:42:42.257069 19040 webserver.cc:460] Webserver started at http://127.0.98.62:46639/ using document root <none> and password file <none>
I20260927 02:42:42.257596 19040 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:42.257670 19040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:42.257850 19040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:42.259943 19040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/master-0/data/instance:
uuid: "c4597c934ad74b50969bdf329fbd3783"
format_stamp: "Formatted at 2026-09-27 02:42:42 on dist-test-slave-zv8z"
server_key: "6054ea28d44776623fdbc168a9e52293"
server_key_iv: "8eb0dcfa26d55271c8682dc3c48fe488"
server_key_version: "encryptionkey@0"
I20260927 02:42:42.260622 19040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/master-0/wal/instance:
uuid: "c4597c934ad74b50969bdf329fbd3783"
format_stamp: "Formatted at 2026-09-27 02:42:42 on dist-test-slave-zv8z"
server_key: "6054ea28d44776623fdbc168a9e52293"
server_key_iv: "8eb0dcfa26d55271c8682dc3c48fe488"
server_key_version: "encryptionkey@0"
I20260927 02:42:42.264389 19040 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 02:42:42.266839 19054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:42.268052 19040 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:42:42.268151 19040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "c4597c934ad74b50969bdf329fbd3783"
format_stamp: "Formatted at 2026-09-27 02:42:42 on dist-test-slave-zv8z"
server_key: "6054ea28d44776623fdbc168a9e52293"
server_key_iv: "8eb0dcfa26d55271c8682dc3c48fe488"
server_key_version: "encryptionkey@0"
I20260927 02:42:42.268276 19040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:42.279042 19040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:42.279628 19040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:42.279781 19040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:42.286144 19040 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36539
I20260927 02:42:42.286157 19106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36539 every 8 connection(s)
I20260927 02:42:42.287029 19040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:42:42.289963 19107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:42.293341 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 19040
I20260927 02:42:42.293462 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/master-0/wal/instance
I20260927 02:42:42.293769 392 external_mini_cluster.cc:1428] Setting key 4a7ec002fe6d5c4815f1eb4283cf08b9
I20260927 02:42:42.295637 19107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4597c934ad74b50969bdf329fbd3783: Bootstrap starting.
I20260927 02:42:42.297724 19107 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4597c934ad74b50969bdf329fbd3783: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:42.298435 19107 log.cc:824] T 00000000000000000000000000000000 P c4597c934ad74b50969bdf329fbd3783: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:42.300096 19107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4597c934ad74b50969bdf329fbd3783: No bootstrap required, opened a new log
I20260927 02:42:42.302556 19110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:42.295113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40158 (local address 127.0.98.62:36539)
0927 02:42:42.295743 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:42.295761 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:42.295804 (+ 43us) server_negotiation.cc:407] Connection header received
0927 02:42:42.296438 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:42.296454 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:42.296828 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:42.297127 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:42.297542 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:42.298567 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:42.299413 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:42.299868 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:42.300004 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:42.300773 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:42.300778 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:42.300790 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:42:42.301156 (+ 366us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:42.301207 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:42:42.301225 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:42.301308 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:42.301441 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 02:42:42.301674 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":126,"threads_started":1}
I20260927 02:42:42.303048 19107 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4597c934ad74b50969bdf329fbd3783 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4597c934ad74b50969bdf329fbd3783" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36539 } }
I20260927 02:42:42.303258 19107 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4597c934ad74b50969bdf329fbd3783 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:42.303331 19107 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4597c934ad74b50969bdf329fbd3783 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4597c934ad74b50969bdf329fbd3783, State: Initialized, Role: FOLLOWER
I20260927 02:42:42.303721 19107 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4597c934ad74b50969bdf329fbd3783 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4597c934ad74b50969bdf329fbd3783" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36539 } }
I20260927 02:42:42.303836 19107 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4597c934ad74b50969bdf329fbd3783 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:42:42.303879 19107 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4597c934ad74b50969bdf329fbd3783 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:42:42.303944 19107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4597c934ad74b50969bdf329fbd3783 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:42.304775 19107 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4597c934ad74b50969bdf329fbd3783 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4597c934ad74b50969bdf329fbd3783" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36539 } }
I20260927 02:42:42.305049 19107 leader_election.cc:302] T 00000000000000000000000000000000 P c4597c934ad74b50969bdf329fbd3783 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4597c934ad74b50969bdf329fbd3783; no voters:
I20260927 02:42:42.305274 19107 leader_election.cc:288] T 00000000000000000000000000000000 P c4597c934ad74b50969bdf329fbd3783 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:42:42.305452 19112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4597c934ad74b50969bdf329fbd3783 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:42.305707 19112 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4597c934ad74b50969bdf329fbd3783 [term 1 LEADER]: Becoming Leader. State: Replica: c4597c934ad74b50969bdf329fbd3783, State: Running, Role: LEADER
I20260927 02:42:42.306011 19112 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4597c934ad74b50969bdf329fbd3783 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4597c934ad74b50969bdf329fbd3783" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36539 } }
I20260927 02:42:42.306483 19107 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4597c934ad74b50969bdf329fbd3783 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:42:42.307750 19113 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4597c934ad74b50969bdf329fbd3783 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4597c934ad74b50969bdf329fbd3783" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4597c934ad74b50969bdf329fbd3783" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36539 } } }
I20260927 02:42:42.307704 19114 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4597c934ad74b50969bdf329fbd3783 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4597c934ad74b50969bdf329fbd3783. Latest consensus state: current_term: 1 leader_uuid: "c4597c934ad74b50969bdf329fbd3783" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4597c934ad74b50969bdf329fbd3783" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36539 } } }
I20260927 02:42:42.307888 19113 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4597c934ad74b50969bdf329fbd3783 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:42.307891 19114 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4597c934ad74b50969bdf329fbd3783 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:42.308368 19125 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:42:42.311280 19125 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:42:42.316355 19125 catalog_manager.cc:1347] Generated new cluster ID: 28620d4a1ab14cf7b80731704e024c71
I20260927 02:42:42.316416 19125 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:42:42.335599 19125 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:42:42.336848 19125 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:42:42.344309 19125 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4597c934ad74b50969bdf329fbd3783: Generated new TSK 0
I20260927 02:42:42.345033 19125 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:42:42.353093 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36539
--builtin_ntp_servers=127.0.98.20:45489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 02:42:42.461196 19131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:42.461423 19131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:42.461460 19131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:42.465440 19131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:42.465493 19131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:42.465548 19131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:42:42.470165 19131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:42:42.471222 19131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:42.472584 19131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:42.481639 19136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:42.481727 19137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:42.481969 19139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:42.482288 19131 server_base.cc:1034] running on GCE node
I20260927 02:42:42.482699 19131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:42.483309 19131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:42.484469 19131 hybrid_clock.cc:648] HybridClock initialized: now 1790476962484440 us; error 40 us; skew 500 ppm
I20260927 02:42:42.486348 19131 webserver.cc:460] Webserver started at http://127.0.98.1:36131/ using document root <none> and password file <none>
I20260927 02:42:42.486867 19131 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:42.486930 19131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:42.487102 19131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:42.489059 19131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/ts-0/data/instance:
uuid: "b26a8e7acb4c4697ae81db502517fedc"
format_stamp: "Formatted at 2026-09-27 02:42:42 on dist-test-slave-zv8z"
server_key: "0267deb859ac1962efe24b09f2d0077b"
server_key_iv: "a2ac78c068dbd8f293bb5319b5d3653c"
server_key_version: "encryptionkey@0"
I20260927 02:42:42.489567 19131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b26a8e7acb4c4697ae81db502517fedc"
format_stamp: "Formatted at 2026-09-27 02:42:42 on dist-test-slave-zv8z"
server_key: "0267deb859ac1962efe24b09f2d0077b"
server_key_iv: "a2ac78c068dbd8f293bb5319b5d3653c"
server_key_version: "encryptionkey@0"
I20260927 02:42:42.493007 19131 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260927 02:42:42.495245 19145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:42.496402 19131 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:42:42.496510 19131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/ts-0/wal
uuid: "b26a8e7acb4c4697ae81db502517fedc"
format_stamp: "Formatted at 2026-09-27 02:42:42 on dist-test-slave-zv8z"
server_key: "0267deb859ac1962efe24b09f2d0077b"
server_key_iv: "a2ac78c068dbd8f293bb5319b5d3653c"
server_key_version: "encryptionkey@0"
I20260927 02:42:42.496600 19131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:42.511333 19131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:42.511929 19131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:42.512091 19131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:42.512627 19131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:42.513609 19131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:42.513664 19131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:42.513713 19131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:42.513741 19131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:42.522197 19131 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38823
I20260927 02:42:42.522214 19258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38823 every 8 connection(s)
I20260927 02:42:42.523028 19131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:42:42.529009 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 19131
I20260927 02:42:42.529124 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:42:42.529376 392 external_mini_cluster.cc:1428] Setting key 284df49273863348c5c86123d8fa2d51
I20260927 02:42:42.531262 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36539
--builtin_ntp_servers=127.0.98.20:45489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 02:42:42.532038 19110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:42.524772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54197 (local address 127.0.98.62:36539)
0927 02:42:42.524889 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:42.524893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:42.526023 (+ 1130us) server_negotiation.cc:407] Connection header received
0927 02:42:42.527398 (+ 1375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:42.527402 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:42.527460 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:42.527532 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:42.528421 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:42.528956 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:42.530125 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:42.530394 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:42.530484 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:42.531596 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:42.531600 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:42.531603 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:42:42.531616 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:42.531637 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:42:42.531645 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:42.531705 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:42.531841 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 02:42:42.531900 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 02:42:42.532893 19261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:42.525133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.1:54197)
0927 02:42:42.525863 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:42.525905 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:42.527187 (+ 1282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:42.527719 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:42.527726 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:42.527811 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:42.528282 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:42.528294 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:42.529075 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:42.529077 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:42.530003 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:42.530013 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:42.530529 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:42.530533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:42.530702 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:42.531284 (+ 582us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:42:42.531286 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:42.531327 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:42:42.531413 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:42.531715 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:42.531725 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:42.531730 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:42:42.531836 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 02:42:42.532106 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":112,"threads_started":1}
I20260927 02:42:42.534162 19259 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36539
I20260927 02:42:42.534394 19259 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:42.534917 19259 heartbeater.cc:511] Master 127.0.98.62:36539 requested a full tablet report, sending...
I20260927 02:42:42.536561 19071 ts_manager.cc:194] Registered new tserver with Master: b26a8e7acb4c4697ae81db502517fedc (127.0.98.1:38823)
I20260927 02:42:42.538116 19071 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:54197
W20260927 02:42:42.638228 19262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:42.638490 19262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:42.638525 19262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:42.642416 19262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:42.642463 19262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:42.642515 19262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:42:42.647104 19262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:42:42.648037 19262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:42.649296 19262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:42.658460 19268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:42.658460 19270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:42.658466 19267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:42.658522 19262 server_base.cc:1034] running on GCE node
I20260927 02:42:42.659073 19262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:42.659660 19262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:42.660853 19262 hybrid_clock.cc:648] HybridClock initialized: now 1790476962660832 us; error 34 us; skew 500 ppm
I20260927 02:42:42.662571 19262 webserver.cc:460] Webserver started at http://127.0.98.2:41167/ using document root <none> and password file <none>
I20260927 02:42:42.663147 19262 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:42.663208 19262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:42.663385 19262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:42.665295 19262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/ts-1/data/instance:
uuid: "cd9e77e4176f432bb694090c2eb7b853"
format_stamp: "Formatted at 2026-09-27 02:42:42 on dist-test-slave-zv8z"
server_key: "da6464fdc1a1abc5bc2625e7b736bd2e"
server_key_iv: "9a0728483e30b838ac070087c6e8a7d9"
server_key_version: "encryptionkey@0"
I20260927 02:42:42.665804 19262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/ts-1/wal/instance:
uuid: "cd9e77e4176f432bb694090c2eb7b853"
format_stamp: "Formatted at 2026-09-27 02:42:42 on dist-test-slave-zv8z"
server_key: "da6464fdc1a1abc5bc2625e7b736bd2e"
server_key_iv: "9a0728483e30b838ac070087c6e8a7d9"
server_key_version: "encryptionkey@0"
I20260927 02:42:42.669245 19262 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.001s
I20260927 02:42:42.671551 19276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:42.672979 19262 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:42:42.673082 19262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/ts-1/wal
uuid: "cd9e77e4176f432bb694090c2eb7b853"
format_stamp: "Formatted at 2026-09-27 02:42:42 on dist-test-slave-zv8z"
server_key: "da6464fdc1a1abc5bc2625e7b736bd2e"
server_key_iv: "9a0728483e30b838ac070087c6e8a7d9"
server_key_version: "encryptionkey@0"
I20260927 02:42:42.673170 19262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:42.688495 19262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:42.689126 19262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:42.689291 19262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:42.689826 19262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:42.690773 19262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:42.690819 19262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:42.690868 19262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:42.690894 19262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:42.699105 19262 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46145
I20260927 02:42:42.699139 19389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46145 every 8 connection(s)
I20260927 02:42:42.699996 19262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:42:42.707391 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 19262
I20260927 02:42:42.707525 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:42:42.707856 392 external_mini_cluster.cc:1428] Setting key f04e4ed7eb8b81ef960c0fcd9d1c9704
I20260927 02:42:42.709198 19110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:42.701747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58077 (local address 127.0.98.62:36539)
0927 02:42:42.701867 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:42.701871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:42.702941 (+ 1070us) server_negotiation.cc:407] Connection header received
0927 02:42:42.704272 (+ 1331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:42.704275 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:42.704329 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:42.704455 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:42.705311 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:42.705865 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:42.707164 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:42.707491 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:42.707604 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:42.708764 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:42.708767 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:42.708769 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:42:42.708780 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:42.708799 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:42:42.708807 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:42.708862 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:42.709013 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 02:42:42.709071 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 02:42:42.709964 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36539
--builtin_ntp_servers=127.0.98.20:45489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 02:42:42.710027 19392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:42.702103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.2:58077)
0927 02:42:42.702798 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:42.702837 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:42.704048 (+ 1211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:42.704594 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:42.704600 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:42.704681 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:42.705154 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:42.705166 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:42.705977 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:42.705980 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:42.707030 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:42.707040 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:42.707678 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:42.707683 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:42.707898 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:42.708499 (+ 601us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:42:42.708501 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:42.708542 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:42:42.708621 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:42.708880 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:42.708887 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:42.708892 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:42:42.709015 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 02:42:42.709324 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":149,"threads_started":1}
I20260927 02:42:42.711186 19390 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36539
I20260927 02:42:42.711422 19390 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:42.711990 19390 heartbeater.cc:511] Master 127.0.98.62:36539 requested a full tablet report, sending...
I20260927 02:42:42.713110 19071 ts_manager.cc:194] Registered new tserver with Master: cd9e77e4176f432bb694090c2eb7b853 (127.0.98.2:46145)
I20260927 02:42:42.713650 19071 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:58077
W20260927 02:42:42.815707 19393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:42.815933 19393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:42.815981 19393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:42.819906 19393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:42.819952 19393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:42.820020 19393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:42:42.824569 19393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:42:42.825510 19393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:42.826836 19393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:42.836026 19399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:42.836128 19401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:42.836033 19398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:42.836226 19393 server_base.cc:1034] running on GCE node
I20260927 02:42:42.836649 19393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:42.837301 19393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:42.838467 19393 hybrid_clock.cc:648] HybridClock initialized: now 1790476962838450 us; error 30 us; skew 500 ppm
I20260927 02:42:42.840121 19393 webserver.cc:460] Webserver started at http://127.0.98.3:42129/ using document root <none> and password file <none>
I20260927 02:42:42.840680 19393 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:42.840745 19393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:42.840934 19393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:42.842898 19393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/ts-2/data/instance:
uuid: "57e23839c54c4a3ca79e930735d52ccd"
format_stamp: "Formatted at 2026-09-27 02:42:42 on dist-test-slave-zv8z"
server_key: "3f998dbfd2e400d1780e8eef51222cd2"
server_key_iv: "39d4b5a381630c95633940a0a12c22dd"
server_key_version: "encryptionkey@0"
I20260927 02:42:42.843402 19393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/ts-2/wal/instance:
uuid: "57e23839c54c4a3ca79e930735d52ccd"
format_stamp: "Formatted at 2026-09-27 02:42:42 on dist-test-slave-zv8z"
server_key: "3f998dbfd2e400d1780e8eef51222cd2"
server_key_iv: "39d4b5a381630c95633940a0a12c22dd"
server_key_version: "encryptionkey@0"
I20260927 02:42:42.847045 19393 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:42:42.849421 19407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:42.850710 19393 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:42:42.850852 19393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/ts-2/wal
uuid: "57e23839c54c4a3ca79e930735d52ccd"
format_stamp: "Formatted at 2026-09-27 02:42:42 on dist-test-slave-zv8z"
server_key: "3f998dbfd2e400d1780e8eef51222cd2"
server_key_iv: "39d4b5a381630c95633940a0a12c22dd"
server_key_version: "encryptionkey@0"
I20260927 02:42:42.850956 19393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:42.871616 19393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:42.872265 19393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:42.872424 19393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:42.872958 19393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:42.873835 19393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:42.873878 19393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:42.873929 19393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:42.873955 19393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:42.882078 19393 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35757
I20260927 02:42:42.882104 19520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35757 every 8 connection(s)
I20260927 02:42:42.882939 19393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:42:42.886263 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 19393
I20260927 02:42:42.886369 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:42:42.886610 392 external_mini_cluster.cc:1428] Setting key 15b3a795f8ce2afb5224a4c57b0806f8
I20260927 02:42:42.891480 19110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:42.884601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53881 (local address 127.0.98.62:36539)
0927 02:42:42.884721 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:42.884724 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:42.885702 (+ 978us) server_negotiation.cc:407] Connection header received
0927 02:42:42.887092 (+ 1390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:42.887095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:42.887140 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:42.887210 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:42.888129 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:42.888762 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:42.889834 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:42.890086 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:42.890161 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:42.891106 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:42.891108 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:42.891111 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:42:42.891121 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:42.891139 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:42:42.891153 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:42.891208 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:42.891330 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 02:42:42.891376 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 02:42:42.892226 19523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:42.884958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36539 (local address 127.0.98.3:53881)
0927 02:42:42.885558 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:42.885596 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:42.886891 (+ 1295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:42.887383 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:42.887391 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:42.887491 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:42.888008 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:42.888022 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:42.888861 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:42.888864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:42.889720 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:42.889729 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:42.890192 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:42.890195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:42.890388 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:42.890882 (+ 494us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:42:42.890883 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:42.890920 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:42:42.890979 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:42.891215 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:42.891220 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:42.891224 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:42:42.891318 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 02:42:42.891514 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":106,"threads_started":1}
I20260927 02:42:42.893142 19521 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36539
I20260927 02:42:42.893352 19521 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:42.893810 19521 heartbeater.cc:511] Master 127.0.98.62:36539 requested a full tablet report, sending...
I20260927 02:42:42.894729 19071 ts_manager.cc:194] Registered new tserver with Master: 57e23839c54c4a3ca79e930735d52ccd (127.0.98.3:35757)
I20260927 02:42:42.895275 19071 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:53881
I20260927 02:42:42.900022 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:42:42.905201 19110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:42.901811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40160 (local address 127.0.98.62:36539)
0927 02:42:42.901914 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:42.901917 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:42.902081 (+ 164us) server_negotiation.cc:407] Connection header received
0927 02:42:42.902184 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:42.902186 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:42.902221 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:42.902310 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:42.902528 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:42.903027 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:42.903793 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:42.903987 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:42.904051 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:42.904797 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:42.904800 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:42.904802 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:42:42.904813 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:42.904832 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:42:42.904839 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:42.904940 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:42.905052 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 02:42:42.905084 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 02:42:42.907943 19071 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40160:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:42:42.910863 19071 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:42:42.923540 19532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:42.918358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35757 (local address 127.0.0.1:51602)
0927 02:42:42.918645 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:42.918688 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:42.918799 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:42.919731 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:42.919738 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:42.919760 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:42.919888 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:42.919895 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:42.920969 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:42.920972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:42.921988 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:42.921997 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:42.923030 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:42.923033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:42.923147 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:42.923195 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 02:42:42.923298 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 02:42:42.923405 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":64,"threads_started":1}
I20260927 02:42:42.923579 19533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:42.918488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51602 (local address 127.0.98.3:35757)
0927 02:42:42.918977 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:42.918989 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:42.919013 (+ 24us) server_negotiation.cc:407] Connection header received
0927 02:42:42.919158 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:42.919165 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:42.919372 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:42.919528 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:42.920004 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:42.920876 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:42.922103 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:42.922917 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:42.923020 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:42.923202 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:42.923332 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 02:42:42.923423 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":180,"threads_started":1}
I20260927 02:42:42.924286 19534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:42.918680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38823 (local address 127.0.0.1:47642)
0927 02:42:42.919256 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:42.919270 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:42.919351 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:42.920263 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:42.920266 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:42.920284 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:42.920369 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:42.920374 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:42.921730 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:42.921733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:42.922763 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:42.922772 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:42.923961 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:42.923963 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:42.924025 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:42.924041 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 02:42:42.924102 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 02:42:42.924164 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":1,"thread_start_us":74,"threads_started":1}
I20260927 02:42:42.924398 19536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:42.919091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47642 (local address 127.0.98.1:38823)
0927 02:42:42.919745 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:42.919769 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:42.919792 (+ 23us) server_negotiation.cc:407] Connection header received
0927 02:42:42.919865 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:42.919870 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:42.920057 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:42.920275 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:42.920474 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:42.921615 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:42.922901 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:42.923632 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:42.923741 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:42.923904 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:42.924151 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 02:42:42.924252 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":149,"threads_started":1}
I20260927 02:42:42.924672 19535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:42.919044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46145 (local address 127.0.0.1:42916)
0927 02:42:42.919416 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:42.919429 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:42.919509 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:42.920366 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:42.920369 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:42.920386 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:42.920468 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:42.920473 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:42.921850 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:42.921854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:42.923017 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:42.923025 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:42.924336 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:42.924340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:42.924424 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:42.924446 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 02:42:42.924492 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 02:42:42.924554 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":86,"threads_started":1}
I20260927 02:42:42.925297 19537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:42.919584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42916 (local address 127.0.98.2:46145)
0927 02:42:42.919969 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:42.919978 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:42.919997 (+ 19us) server_negotiation.cc:407] Connection header received
0927 02:42:42.920057 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:42.920061 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:42.920222 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:42.920611 (+ 389us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:42.920732 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:42.921762 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:42.923137 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:42.924204 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:42.924860 (+ 656us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:42.925022 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:42.925068 (+ 46us) server_negotiation.cc:299] Negotiation successful
0927 02:42:42.925150 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":125,"threads_started":1}
I20260927 02:42:42.927320 19455 tablet_service.cc:1467] Processing CreateTablet for tablet 2b8b6cbc37ac4a40b412e989316ffb76 (DEFAULT_TABLE table=test-table [id=4b500b61598f4e3e9381e963d6e891d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:42.928409 19455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b8b6cbc37ac4a40b412e989316ffb76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:42.928012 19324 tablet_service.cc:1467] Processing CreateTablet for tablet 2b8b6cbc37ac4a40b412e989316ffb76 (DEFAULT_TABLE table=test-table [id=4b500b61598f4e3e9381e963d6e891d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:42.928308 19193 tablet_service.cc:1467] Processing CreateTablet for tablet 2b8b6cbc37ac4a40b412e989316ffb76 (DEFAULT_TABLE table=test-table [id=4b500b61598f4e3e9381e963d6e891d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:42.928984 19324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b8b6cbc37ac4a40b412e989316ffb76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:42.929219 19193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b8b6cbc37ac4a40b412e989316ffb76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:42.935091 19538 tablet_bootstrap.cc:492] T 2b8b6cbc37ac4a40b412e989316ffb76 P b26a8e7acb4c4697ae81db502517fedc: Bootstrap starting.
I20260927 02:42:42.936321 19540 tablet_bootstrap.cc:492] T 2b8b6cbc37ac4a40b412e989316ffb76 P cd9e77e4176f432bb694090c2eb7b853: Bootstrap starting.
I20260927 02:42:42.937091 19538 tablet_bootstrap.cc:654] T 2b8b6cbc37ac4a40b412e989316ffb76 P b26a8e7acb4c4697ae81db502517fedc: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:42.937785 19538 log.cc:824] T 2b8b6cbc37ac4a40b412e989316ffb76 P b26a8e7acb4c4697ae81db502517fedc: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:42.937958 19540 tablet_bootstrap.cc:654] T 2b8b6cbc37ac4a40b412e989316ffb76 P cd9e77e4176f432bb694090c2eb7b853: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:42.938606 19539 tablet_bootstrap.cc:492] T 2b8b6cbc37ac4a40b412e989316ffb76 P 57e23839c54c4a3ca79e930735d52ccd: Bootstrap starting.
I20260927 02:42:42.938704 19540 log.cc:824] T 2b8b6cbc37ac4a40b412e989316ffb76 P cd9e77e4176f432bb694090c2eb7b853: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:42.939728 19538 tablet_bootstrap.cc:492] T 2b8b6cbc37ac4a40b412e989316ffb76 P b26a8e7acb4c4697ae81db502517fedc: No bootstrap required, opened a new log
I20260927 02:42:42.940299 19538 ts_tablet_manager.cc:1397] T 2b8b6cbc37ac4a40b412e989316ffb76 P b26a8e7acb4c4697ae81db502517fedc: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260927 02:42:42.940517 19539 tablet_bootstrap.cc:654] T 2b8b6cbc37ac4a40b412e989316ffb76 P 57e23839c54c4a3ca79e930735d52ccd: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:42.940558 19540 tablet_bootstrap.cc:492] T 2b8b6cbc37ac4a40b412e989316ffb76 P cd9e77e4176f432bb694090c2eb7b853: No bootstrap required, opened a new log
I20260927 02:42:42.940719 19540 ts_tablet_manager.cc:1397] T 2b8b6cbc37ac4a40b412e989316ffb76 P cd9e77e4176f432bb694090c2eb7b853: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 02:42:42.941293 19539 log.cc:824] T 2b8b6cbc37ac4a40b412e989316ffb76 P 57e23839c54c4a3ca79e930735d52ccd: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:42.943182 19539 tablet_bootstrap.cc:492] T 2b8b6cbc37ac4a40b412e989316ffb76 P 57e23839c54c4a3ca79e930735d52ccd: No bootstrap required, opened a new log
I20260927 02:42:42.943365 19539 ts_tablet_manager.cc:1397] T 2b8b6cbc37ac4a40b412e989316ffb76 P 57e23839c54c4a3ca79e930735d52ccd: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 02:42:42.943761 19538 raft_consensus.cc:348] T 2b8b6cbc37ac4a40b412e989316ffb76 P b26a8e7acb4c4697ae81db502517fedc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57e23839c54c4a3ca79e930735d52ccd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35757 } } peers { permanent_uuid: "cd9e77e4176f432bb694090c2eb7b853" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46145 } } peers { permanent_uuid: "b26a8e7acb4c4697ae81db502517fedc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38823 } }
I20260927 02:42:42.943964 19538 raft_consensus.cc:374] T 2b8b6cbc37ac4a40b412e989316ffb76 P b26a8e7acb4c4697ae81db502517fedc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:42.944038 19538 raft_consensus.cc:729] T 2b8b6cbc37ac4a40b412e989316ffb76 P b26a8e7acb4c4697ae81db502517fedc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b26a8e7acb4c4697ae81db502517fedc, State: Initialized, Role: FOLLOWER
I20260927 02:42:42.944440 19538 consensus_queue.cc:260] T 2b8b6cbc37ac4a40b412e989316ffb76 P b26a8e7acb4c4697ae81db502517fedc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57e23839c54c4a3ca79e930735d52ccd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35757 } } peers { permanent_uuid: "cd9e77e4176f432bb694090c2eb7b853" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46145 } } peers { permanent_uuid: "b26a8e7acb4c4697ae81db502517fedc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38823 } }
I20260927 02:42:42.944639 19540 raft_consensus.cc:348] T 2b8b6cbc37ac4a40b412e989316ffb76 P cd9e77e4176f432bb694090c2eb7b853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57e23839c54c4a3ca79e930735d52ccd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35757 } } peers { permanent_uuid: "cd9e77e4176f432bb694090c2eb7b853" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46145 } } peers { permanent_uuid: "b26a8e7acb4c4697ae81db502517fedc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38823 } }
I20260927 02:42:42.944936 19540 raft_consensus.cc:374] T 2b8b6cbc37ac4a40b412e989316ffb76 P cd9e77e4176f432bb694090c2eb7b853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:42.945011 19259 heartbeater.cc:503] Master 127.0.98.62:36539 was elected leader, sending a full tablet report...
I20260927 02:42:42.945019 19540 raft_consensus.cc:729] T 2b8b6cbc37ac4a40b412e989316ffb76 P cd9e77e4176f432bb694090c2eb7b853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd9e77e4176f432bb694090c2eb7b853, State: Initialized, Role: FOLLOWER
I20260927 02:42:42.945364 19538 ts_tablet_manager.cc:1428] T 2b8b6cbc37ac4a40b412e989316ffb76 P b26a8e7acb4c4697ae81db502517fedc: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20260927 02:42:42.945468 19540 consensus_queue.cc:260] T 2b8b6cbc37ac4a40b412e989316ffb76 P cd9e77e4176f432bb694090c2eb7b853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57e23839c54c4a3ca79e930735d52ccd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35757 } } peers { permanent_uuid: "cd9e77e4176f432bb694090c2eb7b853" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46145 } } peers { permanent_uuid: "b26a8e7acb4c4697ae81db502517fedc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38823 } }
I20260927 02:42:42.946156 19390 heartbeater.cc:503] Master 127.0.98.62:36539 was elected leader, sending a full tablet report...
I20260927 02:42:42.946450 19540 ts_tablet_manager.cc:1428] T 2b8b6cbc37ac4a40b412e989316ffb76 P cd9e77e4176f432bb694090c2eb7b853: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 02:42:42.947410 19539 raft_consensus.cc:348] T 2b8b6cbc37ac4a40b412e989316ffb76 P 57e23839c54c4a3ca79e930735d52ccd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57e23839c54c4a3ca79e930735d52ccd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35757 } } peers { permanent_uuid: "cd9e77e4176f432bb694090c2eb7b853" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46145 } } peers { permanent_uuid: "b26a8e7acb4c4697ae81db502517fedc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38823 } }
I20260927 02:42:42.947681 19539 raft_consensus.cc:374] T 2b8b6cbc37ac4a40b412e989316ffb76 P 57e23839c54c4a3ca79e930735d52ccd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:42.947741 19539 raft_consensus.cc:729] T 2b8b6cbc37ac4a40b412e989316ffb76 P 57e23839c54c4a3ca79e930735d52ccd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57e23839c54c4a3ca79e930735d52ccd, State: Initialized, Role: FOLLOWER
I20260927 02:42:42.948211 19539 consensus_queue.cc:260] T 2b8b6cbc37ac4a40b412e989316ffb76 P 57e23839c54c4a3ca79e930735d52ccd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57e23839c54c4a3ca79e930735d52ccd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35757 } } peers { permanent_uuid: "cd9e77e4176f432bb694090c2eb7b853" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46145 } } peers { permanent_uuid: "b26a8e7acb4c4697ae81db502517fedc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38823 } }
I20260927 02:42:42.949023 19539 ts_tablet_manager.cc:1428] T 2b8b6cbc37ac4a40b412e989316ffb76 P 57e23839c54c4a3ca79e930735d52ccd: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 02:42:42.949349 19521 heartbeater.cc:503] Master 127.0.98.62:36539 was elected leader, sending a full tablet report...
W20260927 02:42:42.950870 19391 tablet.cc:2369] T 2b8b6cbc37ac4a40b412e989316ffb76 P cd9e77e4176f432bb694090c2eb7b853: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:42:42.983799 19544 raft_consensus.cc:482] T 2b8b6cbc37ac4a40b412e989316ffb76 P b26a8e7acb4c4697ae81db502517fedc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:42:42.983994 19544 raft_consensus.cc:504] T 2b8b6cbc37ac4a40b412e989316ffb76 P b26a8e7acb4c4697ae81db502517fedc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57e23839c54c4a3ca79e930735d52ccd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35757 } } peers { permanent_uuid: "cd9e77e4176f432bb694090c2eb7b853" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46145 } } peers { permanent_uuid: "b26a8e7acb4c4697ae81db502517fedc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38823 } }
I20260927 02:42:42.985229 19544 leader_election.cc:288] T 2b8b6cbc37ac4a40b412e989316ffb76 P b26a8e7acb4c4697ae81db502517fedc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 57e23839c54c4a3ca79e930735d52ccd (127.0.98.3:35757), cd9e77e4176f432bb694090c2eb7b853 (127.0.98.2:46145)
I20260927 02:42:42.989204 19533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:42.985613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35585 (local address 127.0.98.3:35757)
0927 02:42:42.985799 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:42.985803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:42.985819 (+ 16us) server_negotiation.cc:407] Connection header received
0927 02:42:42.985876 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:42.985879 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:42.985926 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:42.986001 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:42.986350 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:42.986902 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:42.988097 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:42.988686 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:42.988771 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:42.988819 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:42.989030 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 02:42:42.989085 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260927 02:42:42.989207 19261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:42.985457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35757 (local address 127.0.98.1:35585)
0927 02:42:42.985620 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:42.985637 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:42.985739 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:42.986088 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:42.986091 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:42.986111 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:42.986214 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:42.986220 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:42.987006 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:42.987009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:42.987965 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:42.987974 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:42.988805 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:42.988809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:42.988912 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:42.988937 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 02:42:42.989014 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 02:42:42.989082 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 02:42:42.989869 19547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:42.985781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46145 (local address 127.0.98.1:54661)
0927 02:42:42.986183 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:42.986194 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:42.986304 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:42.986567 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:42.986570 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:42.986586 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:42.986696 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:42.986701 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:42.987567 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:42.987570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:42.988594 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:42.988604 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:42.989499 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:42.989502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:42.989577 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:42.989593 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 02:42:42.989652 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 02:42:42.989739 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":149,"threads_started":1}
I20260927 02:42:42.989894 19537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:42.985932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54661 (local address 127.0.98.2:46145)
0927 02:42:42.986050 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:42.986054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:42.986271 (+ 217us) server_negotiation.cc:407] Connection header received
0927 02:42:42.986420 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:42.986424 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:42.986477 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:42.986541 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:42.986793 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:42.987451 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:42.988732 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:42.989294 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:42.989390 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:42.989443 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:42.989676 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 02:42:42.989744 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 02:42:42.989872 19475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b8b6cbc37ac4a40b412e989316ffb76" candidate_uuid: "b26a8e7acb4c4697ae81db502517fedc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57e23839c54c4a3ca79e930735d52ccd" is_pre_election: true
I20260927 02:42:42.990185 19475 raft_consensus.cc:2454] T 2b8b6cbc37ac4a40b412e989316ffb76 P 57e23839c54c4a3ca79e930735d52ccd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b26a8e7acb4c4697ae81db502517fedc in term 0.
I20260927 02:42:42.990454 19344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b8b6cbc37ac4a40b412e989316ffb76" candidate_uuid: "b26a8e7acb4c4697ae81db502517fedc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd9e77e4176f432bb694090c2eb7b853" is_pre_election: true
I20260927 02:42:42.990657 19146 leader_election.cc:302] T 2b8b6cbc37ac4a40b412e989316ffb76 P b26a8e7acb4c4697ae81db502517fedc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57e23839c54c4a3ca79e930735d52ccd, b26a8e7acb4c4697ae81db502517fedc; no voters:
I20260927 02:42:42.990736 19344 raft_consensus.cc:2454] T 2b8b6cbc37ac4a40b412e989316ffb76 P cd9e77e4176f432bb694090c2eb7b853 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b26a8e7acb4c4697ae81db502517fedc in term 0.
I20260927 02:42:42.990921 19544 raft_consensus.cc:2781] T 2b8b6cbc37ac4a40b412e989316ffb76 P b26a8e7acb4c4697ae81db502517fedc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:42:42.991001 19544 raft_consensus.cc:482] T 2b8b6cbc37ac4a40b412e989316ffb76 P b26a8e7acb4c4697ae81db502517fedc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:42:42.991051 19544 raft_consensus.cc:3037] T 2b8b6cbc37ac4a40b412e989316ffb76 P b26a8e7acb4c4697ae81db502517fedc [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:42.992198 19544 raft_consensus.cc:504] T 2b8b6cbc37ac4a40b412e989316ffb76 P b26a8e7acb4c4697ae81db502517fedc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57e23839c54c4a3ca79e930735d52ccd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35757 } } peers { permanent_uuid: "cd9e77e4176f432bb694090c2eb7b853" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46145 } } peers { permanent_uuid: "b26a8e7acb4c4697ae81db502517fedc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38823 } }
I20260927 02:42:42.992532 19544 leader_election.cc:288] T 2b8b6cbc37ac4a40b412e989316ffb76 P b26a8e7acb4c4697ae81db502517fedc [CANDIDATE]: Term 1 election: Requested vote from peers 57e23839c54c4a3ca79e930735d52ccd (127.0.98.3:35757), cd9e77e4176f432bb694090c2eb7b853 (127.0.98.2:46145)
I20260927 02:42:42.992952 19344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b8b6cbc37ac4a40b412e989316ffb76" candidate_uuid: "b26a8e7acb4c4697ae81db502517fedc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd9e77e4176f432bb694090c2eb7b853"
I20260927 02:42:42.992940 19475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b8b6cbc37ac4a40b412e989316ffb76" candidate_uuid: "b26a8e7acb4c4697ae81db502517fedc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57e23839c54c4a3ca79e930735d52ccd"
I20260927 02:42:42.993067 19475 raft_consensus.cc:3037] T 2b8b6cbc37ac4a40b412e989316ffb76 P 57e23839c54c4a3ca79e930735d52ccd [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:42.993072 19344 raft_consensus.cc:3037] T 2b8b6cbc37ac4a40b412e989316ffb76 P cd9e77e4176f432bb694090c2eb7b853 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:42.994324 19475 raft_consensus.cc:2454] T 2b8b6cbc37ac4a40b412e989316ffb76 P 57e23839c54c4a3ca79e930735d52ccd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b26a8e7acb4c4697ae81db502517fedc in term 1.
I20260927 02:42:42.994333 19344 raft_consensus.cc:2454] T 2b8b6cbc37ac4a40b412e989316ffb76 P cd9e77e4176f432bb694090c2eb7b853 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b26a8e7acb4c4697ae81db502517fedc in term 1.
I20260927 02:42:42.994616 19146 leader_election.cc:302] T 2b8b6cbc37ac4a40b412e989316ffb76 P b26a8e7acb4c4697ae81db502517fedc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b26a8e7acb4c4697ae81db502517fedc, cd9e77e4176f432bb694090c2eb7b853; no voters:
I20260927 02:42:42.994903 19544 raft_consensus.cc:2781] T 2b8b6cbc37ac4a40b412e989316ffb76 P b26a8e7acb4c4697ae81db502517fedc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:42.995096 19544 raft_consensus.cc:686] T 2b8b6cbc37ac4a40b412e989316ffb76 P b26a8e7acb4c4697ae81db502517fedc [term 1 LEADER]: Becoming Leader. State: Replica: b26a8e7acb4c4697ae81db502517fedc, State: Running, Role: LEADER
I20260927 02:42:42.995394 19544 consensus_queue.cc:237] T 2b8b6cbc37ac4a40b412e989316ffb76 P b26a8e7acb4c4697ae81db502517fedc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57e23839c54c4a3ca79e930735d52ccd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35757 } } peers { permanent_uuid: "cd9e77e4176f432bb694090c2eb7b853" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46145 } } peers { permanent_uuid: "b26a8e7acb4c4697ae81db502517fedc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38823 } }
I20260927 02:42:42.998446 19071 catalog_manager.cc:5495] T 2b8b6cbc37ac4a40b412e989316ffb76 P b26a8e7acb4c4697ae81db502517fedc reported cstate change: term changed from 0 to 1, leader changed from <none> to b26a8e7acb4c4697ae81db502517fedc (127.0.98.1). New cstate: current_term: 1 leader_uuid: "b26a8e7acb4c4697ae81db502517fedc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57e23839c54c4a3ca79e930735d52ccd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd9e77e4176f432bb694090c2eb7b853" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b26a8e7acb4c4697ae81db502517fedc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38823 } health_report { overall_health: HEALTHY } } }
I20260927 02:42:43.021910 19536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:43.018768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47658 (local address 127.0.98.1:38823)
0927 02:42:43.018886 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:43.018890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:43.018903 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:42:43.018987 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:43.018990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:43.019041 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:43.019127 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:42:43.019292 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.019775 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.020569 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:43.020829 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:43.020910 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:43.021130 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:42:43.021551 (+ 421us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:42:43.021669 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:43.021743 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 02:42:43.021772 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 02:42:43.024084 19260 tablet.cc:2369] T 2b8b6cbc37ac4a40b412e989316ffb76 P b26a8e7acb4c4697ae81db502517fedc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:42:43.027470 19344 raft_consensus.cc:1261] T 2b8b6cbc37ac4a40b412e989316ffb76 P cd9e77e4176f432bb694090c2eb7b853 [term 1 FOLLOWER]: Refusing update from remote peer b26a8e7acb4c4697ae81db502517fedc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:42:43.027474 19475 raft_consensus.cc:1261] T 2b8b6cbc37ac4a40b412e989316ffb76 P 57e23839c54c4a3ca79e930735d52ccd [term 1 FOLLOWER]: Refusing update from remote peer b26a8e7acb4c4697ae81db502517fedc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:42:43.028132 19544 consensus_queue.cc:1035] T 2b8b6cbc37ac4a40b412e989316ffb76 P b26a8e7acb4c4697ae81db502517fedc [LEADER]: Connected to new peer: Peer: permanent_uuid: "57e23839c54c4a3ca79e930735d52ccd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:42:43.028302 19548 consensus_queue.cc:1035] T 2b8b6cbc37ac4a40b412e989316ffb76 P b26a8e7acb4c4697ae81db502517fedc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd9e77e4176f432bb694090c2eb7b853" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:42:43.036142 19553 mvcc.cc:205] Tried to move back new op lower bound from 7333793640553578496 to 7333793640431767552. Current Snapshot: MvccSnapshot[applied={T|T < 7333793640553578496}]
I20260927 02:42:43.037437 19552 mvcc.cc:205] Tried to move back new op lower bound from 7333793640553578496 to 7333793640431767552. Current Snapshot: MvccSnapshot[applied={T|T < 7333793640553578496}]
I20260927 02:42:43.043589 19069 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:42:43.043737 19069 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:42:43.046036 19069 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c4597c934ad74b50969bdf329fbd3783: Sending DeleteTablet for 3 replicas of tablet 2b8b6cbc37ac4a40b412e989316ffb76
I20260927 02:42:43.046958 19455 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b8b6cbc37ac4a40b412e989316ffb76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:43 UTC) from {username='slave'} at 127.0.0.1:51602
I20260927 02:42:43.046948 19324 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b8b6cbc37ac4a40b412e989316ffb76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:43 UTC) from {username='slave'} at 127.0.0.1:42916
I20260927 02:42:43.046945 19193 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b8b6cbc37ac4a40b412e989316ffb76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:43 UTC) from {username='slave'} at 127.0.0.1:47642
I20260927 02:42:43.047569 19562 tablet_replica.cc:330] T 2b8b6cbc37ac4a40b412e989316ffb76 P 57e23839c54c4a3ca79e930735d52ccd: stopping tablet replica
I20260927 02:42:43.047717 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 19131
I20260927 02:42:43.047917 19562 raft_consensus.cc:2229] T 2b8b6cbc37ac4a40b412e989316ffb76 P 57e23839c54c4a3ca79e930735d52ccd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:42:43.048228 19562 raft_consensus.cc:2258] T 2b8b6cbc37ac4a40b412e989316ffb76 P 57e23839c54c4a3ca79e930735d52ccd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:43.048548 19563 tablet_replica.cc:330] T 2b8b6cbc37ac4a40b412e989316ffb76 P cd9e77e4176f432bb694090c2eb7b853: stopping tablet replica
I20260927 02:42:43.048816 19563 raft_consensus.cc:2229] T 2b8b6cbc37ac4a40b412e989316ffb76 P cd9e77e4176f432bb694090c2eb7b853 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:42:43.049068 19563 raft_consensus.cc:2258] T 2b8b6cbc37ac4a40b412e989316ffb76 P cd9e77e4176f432bb694090c2eb7b853 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:43.049654 19562 ts_tablet_manager.cc:1905] T 2b8b6cbc37ac4a40b412e989316ffb76 P 57e23839c54c4a3ca79e930735d52ccd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:42:43.051918 19562 ts_tablet_manager.cc:1918] T 2b8b6cbc37ac4a40b412e989316ffb76 P 57e23839c54c4a3ca79e930735d52ccd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:42:43.052000 19562 log.cc:1196] T 2b8b6cbc37ac4a40b412e989316ffb76 P 57e23839c54c4a3ca79e930735d52ccd: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790476718023131-392-0/minicluster-data/ts-2/wal/wals/2b8b6cbc37ac4a40b412e989316ffb76
I20260927 02:42:43.052253 19562 ts_tablet_manager.cc:1939] T 2b8b6cbc37ac4a40b412e989316ffb76 P 57e23839c54c4a3ca79e930735d52ccd: Deleting consensus metadata
I20260927 02:42:43.053233 19055 catalog_manager.cc:4841] TS 57e23839c54c4a3ca79e930735d52ccd (127.0.98.3:35757): tablet 2b8b6cbc37ac4a40b412e989316ffb76 (table test-table [id=4b500b61598f4e3e9381e963d6e891d9]) successfully deleted
W20260927 02:42:43.054179 19057 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38823 (error 108)
I20260927 02:42:43.055146 19535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:43.054881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38823 (local address 127.0.0.1:47674)
0927 02:42:43.054996 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:43.055067 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 02:42:43.055212 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 19262
W20260927 02:42:43.055346 19057 catalog_manager.cc:4584] TS b26a8e7acb4c4697ae81db502517fedc (127.0.98.1:38823): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2b8b6cbc37ac4a40b412e989316ffb76: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38823: connect: Connection refused (error 111)
I20260927 02:42:43.060102 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 19393
I20260927 02:42:43.060730 19535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:43.060185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46145 (local address 127.0.0.1:42920)
0927 02:42:43.060625 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:43.060668 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:46145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369}
I20260927 02:42:43.064822 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 19040
2026-09-27T02:42:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (963 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:42:43 dist-test-slave-zv8z krb5kdc[19567](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:42:43 dist-test-slave-zv8z krb5kdc[19567](info): set up 2 sockets
Sep 27 02:42:43 dist-test-slave-zv8z krb5kdc[19567](info): commencing operation
krb5kdc: starting...
W20260927 02:42:44.017938 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/krb5kdc/test-user.keytab.
Sep 27 02:42:44 dist-test-slave-zv8z krb5kdc[19567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:42:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:42:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:42:44.129660 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34409
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34409
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:44.234380 19580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:44.234750 19580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:44.234812 19580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:44.238855 19580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:42:44.238910 19580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:44.238929 19580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:44.238943 19580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:42:44.238958 19580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:42:44.244309 19580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:42:44.245312 19580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:44.246295 19580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:44.254206 19588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:44.254266 19586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:44.254287 19585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:44.254499 19580 server_base.cc:1034] running on GCE node
I20260927 02:42:44.255018 19580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:44.256040 19580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:44.257189 19580 hybrid_clock.cc:648] HybridClock initialized: now 1790476964257176 us; error 29 us; skew 500 ppm
Sep 27 02:42:44 dist-test-slave-zv8z krb5kdc[19567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:44.259745 19580 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:42:44.260533 19580 webserver.cc:460] Webserver started at http://127.0.98.62:39377/ using document root <none> and password file <none>
I20260927 02:42:44.261014 19580 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:44.261067 19580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:44.261215 19580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:44.263109 19580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/master-0/data/instance:
uuid: "b8d6cd4094394687ae1d8f49e733e160"
format_stamp: "Formatted at 2026-09-27 02:42:44 on dist-test-slave-zv8z"
server_key: "4c95efc1ac1a5bfa8c0ae6bb62260fda"
server_key_iv: "e3baeea93b8da1049f7acb9641446093"
server_key_version: "encryptionkey@0"
I20260927 02:42:44.263585 19580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/master-0/wal/instance:
uuid: "b8d6cd4094394687ae1d8f49e733e160"
format_stamp: "Formatted at 2026-09-27 02:42:44 on dist-test-slave-zv8z"
server_key: "4c95efc1ac1a5bfa8c0ae6bb62260fda"
server_key_iv: "e3baeea93b8da1049f7acb9641446093"
server_key_version: "encryptionkey@0"
I20260927 02:42:44.267015 19580 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:42:44.269150 19595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:44.270300 19580 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:42:44.270401 19580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "b8d6cd4094394687ae1d8f49e733e160"
format_stamp: "Formatted at 2026-09-27 02:42:44 on dist-test-slave-zv8z"
server_key: "4c95efc1ac1a5bfa8c0ae6bb62260fda"
server_key_iv: "e3baeea93b8da1049f7acb9641446093"
server_key_version: "encryptionkey@0"
I20260927 02:42:44.270493 19580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:44.289237 19580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:44.292032 19580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:44.292193 19580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:44.298751 19580 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34409
I20260927 02:42:44.298770 19647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34409 every 8 connection(s)
I20260927 02:42:44.299621 19580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:42:44.302222 19648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:44.305040 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 19580
I20260927 02:42:44.305176 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/master-0/wal/instance
I20260927 02:42:44.305446 392 external_mini_cluster.cc:1428] Setting key 66bfc5eb863071d0a620cc91480c25f0
I20260927 02:42:44.307832 19648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8d6cd4094394687ae1d8f49e733e160: Bootstrap starting.
I20260927 02:42:44.310146 19648 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8d6cd4094394687ae1d8f49e733e160: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:44.310997 19648 log.cc:824] T 00000000000000000000000000000000 P b8d6cd4094394687ae1d8f49e733e160: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:44.313036 19648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8d6cd4094394687ae1d8f49e733e160: No bootstrap required, opened a new log
Sep 27 02:42:44 dist-test-slave-zv8z krb5kdc[19567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:42:44.316066 19648 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8d6cd4094394687ae1d8f49e733e160 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8d6cd4094394687ae1d8f49e733e160" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34409 } }
I20260927 02:42:44.316274 19648 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8d6cd4094394687ae1d8f49e733e160 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:44.316346 19648 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8d6cd4094394687ae1d8f49e733e160 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8d6cd4094394687ae1d8f49e733e160, State: Initialized, Role: FOLLOWER
I20260927 02:42:44.316747 19648 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8d6cd4094394687ae1d8f49e733e160 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8d6cd4094394687ae1d8f49e733e160" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34409 } }
I20260927 02:42:44.316866 19648 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8d6cd4094394687ae1d8f49e733e160 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:42:44.316926 19648 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8d6cd4094394687ae1d8f49e733e160 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:42:44.317016 19648 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8d6cd4094394687ae1d8f49e733e160 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:44.317785 19648 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8d6cd4094394687ae1d8f49e733e160 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8d6cd4094394687ae1d8f49e733e160" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34409 } }
I20260927 02:42:44.318079 19648 leader_election.cc:302] T 00000000000000000000000000000000 P b8d6cd4094394687ae1d8f49e733e160 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8d6cd4094394687ae1d8f49e733e160; no voters:
I20260927 02:42:44.318367 19648 leader_election.cc:288] T 00000000000000000000000000000000 P b8d6cd4094394687ae1d8f49e733e160 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:42:44.318431 19653 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8d6cd4094394687ae1d8f49e733e160 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:44.318645 19653 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8d6cd4094394687ae1d8f49e733e160 [term 1 LEADER]: Becoming Leader. State: Replica: b8d6cd4094394687ae1d8f49e733e160, State: Running, Role: LEADER
I20260927 02:42:44.318944 19653 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8d6cd4094394687ae1d8f49e733e160 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8d6cd4094394687ae1d8f49e733e160" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34409 } }
I20260927 02:42:44.319607 19648 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8d6cd4094394687ae1d8f49e733e160 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:42:44.319677 19651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:44.306744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45158 (local address 127.0.98.62:34409)
0927 02:42:44.307137 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:44.307152 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:44.307186 (+ 34us) server_negotiation.cc:407] Connection header received
0927 02:42:44.307775 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:44.307792 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:44.308136 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:44.308414 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:44.309272 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:44.310283 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:44.311381 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:44.311959 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:44.312079 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:44.314636 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:44.314641 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:44.314653 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:44.314678 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:44.316795 (+ 2117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:44.317238 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:44.317243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:44.317247 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:44.317353 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:44.317594 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:44.317597 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:44.317598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:44.317983 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:44.318150 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:44.318473 (+ 323us) server_negotiation.cc:299] Negotiation successful
0927 02:42:44.318929 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":116,"threads_started":1}
I20260927 02:42:44.320070 19653 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8d6cd4094394687ae1d8f49e733e160 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8d6cd4094394687ae1d8f49e733e160. Latest consensus state: current_term: 1 leader_uuid: "b8d6cd4094394687ae1d8f49e733e160" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8d6cd4094394687ae1d8f49e733e160" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34409 } } }
I20260927 02:42:44.320175 19653 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8d6cd4094394687ae1d8f49e733e160 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:44.320544 19654 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8d6cd4094394687ae1d8f49e733e160 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8d6cd4094394687ae1d8f49e733e160" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8d6cd4094394687ae1d8f49e733e160" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34409 } } }
I20260927 02:42:44.320693 19654 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8d6cd4094394687ae1d8f49e733e160 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:44.321766 19658 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:42:44.324380 19658 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:42:44.329247 19658 catalog_manager.cc:1347] Generated new cluster ID: dbfb440c756a4cccadaf4627243bf910
I20260927 02:42:44.329304 19658 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:42:44.343700 19658 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:42:44.344995 19658 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:42:44.354676 19658 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8d6cd4094394687ae1d8f49e733e160: Generated new TSK 0
I20260927 02:42:44.355410 19658 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:42:44.386826 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34409
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:44.492185 19674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:44.492424 19674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:44.492480 19674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:44.496428 19674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:44.496480 19674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:44.496553 19674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:42:44.501520 19674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:42:44.502472 19674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:44.503412 19674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:44.512254 19680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:44.512288 19679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:44.512353 19682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:44.512503 19674 server_base.cc:1034] running on GCE node
I20260927 02:42:44.512872 19674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:44.513447 19674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:44.514678 19674 hybrid_clock.cc:648] HybridClock initialized: now 1790476964514658 us; error 29 us; skew 500 ppm
Sep 27 02:42:44 dist-test-slave-zv8z krb5kdc[19567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:44.517117 19674 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:42:44.517894 19674 webserver.cc:460] Webserver started at http://127.0.98.1:42039/ using document root <none> and password file <none>
I20260927 02:42:44.518409 19674 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:44.518471 19674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:44.518647 19674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:44.520581 19674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/ts-0/data/instance:
uuid: "2251248973ce43b48f5119be4560d5cb"
format_stamp: "Formatted at 2026-09-27 02:42:44 on dist-test-slave-zv8z"
server_key: "416f0cf953f127002162f929a3b2fefd"
server_key_iv: "9f231875ab1eded031208dfb4eda1327"
server_key_version: "encryptionkey@0"
I20260927 02:42:44.521062 19674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2251248973ce43b48f5119be4560d5cb"
format_stamp: "Formatted at 2026-09-27 02:42:44 on dist-test-slave-zv8z"
server_key: "416f0cf953f127002162f929a3b2fefd"
server_key_iv: "9f231875ab1eded031208dfb4eda1327"
server_key_version: "encryptionkey@0"
I20260927 02:42:44.524425 19674 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 02:42:44.526598 19689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:44.527884 19674 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:42:44.527979 19674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/ts-0/wal
uuid: "2251248973ce43b48f5119be4560d5cb"
format_stamp: "Formatted at 2026-09-27 02:42:44 on dist-test-slave-zv8z"
server_key: "416f0cf953f127002162f929a3b2fefd"
server_key_iv: "9f231875ab1eded031208dfb4eda1327"
server_key_version: "encryptionkey@0"
I20260927 02:42:44.528074 19674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:44.555249 19674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:44.558487 19674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:44.558658 19674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:44.559262 19674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:44.560073 19674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:44.560127 19674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:44.560177 19674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:44.560205 19674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:44.568531 19674 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37543
I20260927 02:42:44.568552 19802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37543 every 8 connection(s)
I20260927 02:42:44.569384 19674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:42:44.572970 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 19674
I20260927 02:42:44.573103 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:42:44.573396 392 external_mini_cluster.cc:1428] Setting key 6b4526d379db0d2a0b48d3038998d4d7
Sep 27 02:42:44 dist-test-slave-zv8z krb5kdc[19567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:42:44.583873 19651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:44.571116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59643 (local address 127.0.98.62:34409)
0927 02:42:44.571250 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:44.571254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:44.572039 (+ 785us) server_negotiation.cc:407] Connection header received
0927 02:42:44.573521 (+ 1482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:44.573525 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:44.573575 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:44.573645 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:44.575082 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:44.575604 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:44.577018 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:44.577361 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:44.577473 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:44.580302 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:44.580305 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:44.580307 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:44.580330 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:44.581730 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:44.582222 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:44.582225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:44.582227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:44.582338 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:44.582646 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:44.582650 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:44.582651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:44.582828 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:44.582912 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:44.583504 (+ 592us) server_negotiation.cc:299] Negotiation successful
0927 02:42:44.583711 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 02:42:44.584686 19805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:44.571452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34409 (local address 127.0.98.1:59643)
0927 02:42:44.571895 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:44.571937 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:44.573278 (+ 1341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:44.573845 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:44.573852 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:44.574299 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:44.574911 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:44.574929 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:44.575777 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:44.575780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:44.576887 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:44.576896 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:44.577538 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:44.577541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:44.577733 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:44.578495 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:44.578514 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:44.580132 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:44.581848 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:44.581854 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:44.581869 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:44.582127 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:44.582418 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:44.582421 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:44.582424 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:44.582560 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:44.582924 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:44.582931 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:44.583257 (+ 326us) client_negotiation.cc:769] Sending connection context
0927 02:42:44.583457 (+ 200us) client_negotiation.cc:240] Negotiation successful
0927 02:42:44.583782 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":108,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 02:42:44.585922 19803 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34409
I20260927 02:42:44.586181 19803 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:44.586778 19803 heartbeater.cc:511] Master 127.0.98.62:34409 requested a full tablet report, sending...
I20260927 02:42:44.588361 19611 ts_manager.cc:194] Registered new tserver with Master: 2251248973ce43b48f5119be4560d5cb (127.0.98.1:37543)
I20260927 02:42:44.589843 19611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59643
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:42:44.601037 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34409
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:44.706344 19808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:44.706604 19808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:44.706660 19808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:44.710810 19808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:44.710870 19808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:44.710944 19808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:42:44.716100 19808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:42:44.717061 19808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:44.718035 19808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:44.727211 19816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:44.727231 19813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:44.727252 19814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:44.727300 19808 server_base.cc:1034] running on GCE node
I20260927 02:42:44.727846 19808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:44.728420 19808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:44.729602 19808 hybrid_clock.cc:648] HybridClock initialized: now 1790476964729581 us; error 32 us; skew 500 ppm
Sep 27 02:42:44 dist-test-slave-zv8z krb5kdc[19567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:44.732298 19808 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:42:44.733088 19808 webserver.cc:460] Webserver started at http://127.0.98.2:37233/ using document root <none> and password file <none>
I20260927 02:42:44.733592 19808 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:44.733656 19808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:44.733839 19808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:44.735801 19808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/ts-1/data/instance:
uuid: "1ea2444bf029428cb6fd2c85d752b58e"
format_stamp: "Formatted at 2026-09-27 02:42:44 on dist-test-slave-zv8z"
server_key: "7a015590ff4663e675265140df8731f0"
server_key_iv: "0ae650d5c6f233f9e9631a6330244000"
server_key_version: "encryptionkey@0"
I20260927 02:42:44.736330 19808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1ea2444bf029428cb6fd2c85d752b58e"
format_stamp: "Formatted at 2026-09-27 02:42:44 on dist-test-slave-zv8z"
server_key: "7a015590ff4663e675265140df8731f0"
server_key_iv: "0ae650d5c6f233f9e9631a6330244000"
server_key_version: "encryptionkey@0"
I20260927 02:42:44.739763 19808 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260927 02:42:44.741961 19823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:44.743156 19808 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:42:44.743258 19808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/ts-1/wal
uuid: "1ea2444bf029428cb6fd2c85d752b58e"
format_stamp: "Formatted at 2026-09-27 02:42:44 on dist-test-slave-zv8z"
server_key: "7a015590ff4663e675265140df8731f0"
server_key_iv: "0ae650d5c6f233f9e9631a6330244000"
server_key_version: "encryptionkey@0"
I20260927 02:42:44.743345 19808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:44.760035 19808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:44.762699 19808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:44.762856 19808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:44.763353 19808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:44.764112 19808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:44.764158 19808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:44.764205 19808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:44.764231 19808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:44.772002 19808 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40747
I20260927 02:42:44.772048 19936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40747 every 8 connection(s)
I20260927 02:42:44.772846 19808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:42:44.776623 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 19808
I20260927 02:42:44.776726 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:42:44.776998 392 external_mini_cluster.cc:1428] Setting key 502b7fbad56c49cc5f0c7b6af5ad1bda
Sep 27 02:42:44 dist-test-slave-zv8z krb5kdc[19567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:42:44.786391 19651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:44.774444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40691 (local address 127.0.98.62:34409)
0927 02:42:44.774561 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:44.774566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:44.775383 (+ 817us) server_negotiation.cc:407] Connection header received
0927 02:42:44.776581 (+ 1198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:44.776585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:44.776650 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:44.776739 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:44.778143 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:44.778782 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:44.780295 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:44.780623 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:44.780719 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:44.782914 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:44.782918 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:44.782921 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:44.782949 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:44.784440 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:44.784967 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:44.784971 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:44.784973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:44.785054 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:44.785339 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:44.785343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:44.785345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:44.785556 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:44.785673 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:44.786129 (+ 456us) server_negotiation.cc:299] Negotiation successful
0927 02:42:44.786236 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 02:42:44.787083 19939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:44.774824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34409 (local address 127.0.98.2:40691)
0927 02:42:44.775240 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:44.775280 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:44.776383 (+ 1103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:44.776975 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:44.776984 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:44.777441 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:44.777989 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:44.778009 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:44.778904 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:44.778907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:44.780190 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:44.780199 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:44.780771 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:44.780774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:44.780919 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:44.781423 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:44.781438 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:44.782761 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:44.784558 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:44.784562 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:44.784573 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:44.784871 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:44.785136 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:44.785138 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:44.785139 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:44.785254 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:44.785656 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:44.785661 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:44.785899 (+ 238us) client_negotiation.cc:769] Sending connection context
0927 02:42:44.786057 (+ 158us) client_negotiation.cc:240] Negotiation successful
0927 02:42:44.786333 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":109,"threads_started":1}
I20260927 02:42:44.788233 19937 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34409
I20260927 02:42:44.788457 19937 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 02:42:44.788904 19937 heartbeater.cc:511] Master 127.0.98.62:34409 requested a full tablet report, sending...
I20260927 02:42:44.789901 19611 ts_manager.cc:194] Registered new tserver with Master: 1ea2444bf029428cb6fd2c85d752b58e (127.0.98.2:40747)
I20260927 02:42:44.790488 19611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40691
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:42:44.805776 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34409
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:44.911379 19942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:44.911612 19942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:44.911661 19942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:44.915553 19942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:44.915604 19942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:44.915673 19942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:42:44.920820 19942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:42:44.921790 19942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:44.922755 19942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:44.931761 19948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:44.931783 19947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:44.931887 19942 server_base.cc:1034] running on GCE node
W20260927 02:42:44.931783 19950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:44.932403 19942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:44.932977 19942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:44.934128 19942 hybrid_clock.cc:648] HybridClock initialized: now 1790476964934108 us; error 33 us; skew 500 ppm
Sep 27 02:42:44 dist-test-slave-zv8z krb5kdc[19567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:44.936869 19942 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:42:44.937736 19942 webserver.cc:460] Webserver started at http://127.0.98.3:36459/ using document root <none> and password file <none>
I20260927 02:42:44.938333 19942 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:44.938402 19942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:44.938591 19942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:44.940584 19942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/ts-2/data/instance:
uuid: "5334b3380b0f44c49e866ca1b6910fed"
format_stamp: "Formatted at 2026-09-27 02:42:44 on dist-test-slave-zv8z"
server_key: "c1fb8c3c19cbbbf4905201777690c57c"
server_key_iv: "49943c407438d20826e299b20a8256eb"
server_key_version: "encryptionkey@0"
I20260927 02:42:44.941143 19942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5334b3380b0f44c49e866ca1b6910fed"
format_stamp: "Formatted at 2026-09-27 02:42:44 on dist-test-slave-zv8z"
server_key: "c1fb8c3c19cbbbf4905201777690c57c"
server_key_iv: "49943c407438d20826e299b20a8256eb"
server_key_version: "encryptionkey@0"
I20260927 02:42:44.944787 19942 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:42:44.947247 19957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:44.948474 19942 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:42:44.948578 19942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/ts-2/wal
uuid: "5334b3380b0f44c49e866ca1b6910fed"
format_stamp: "Formatted at 2026-09-27 02:42:44 on dist-test-slave-zv8z"
server_key: "c1fb8c3c19cbbbf4905201777690c57c"
server_key_iv: "49943c407438d20826e299b20a8256eb"
server_key_version: "encryptionkey@0"
I20260927 02:42:44.948674 19942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:44.971141 19942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:44.974391 19942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:44.974568 19942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:44.975255 19942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:44.976074 19942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:44.976121 19942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:44.976173 19942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:44.976202 19942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:44.984984 19942 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42261
I20260927 02:42:44.985014 20070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42261 every 8 connection(s)
I20260927 02:42:44.985838 19942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:42:44.991554 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 19942
I20260927 02:42:44.991645 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:42:44.991885 392 external_mini_cluster.cc:1428] Setting key ebd1a61633e191deba782b5d5cbaef56
Sep 27 02:42:44 dist-test-slave-zv8z krb5kdc[19567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:42:44.998900 19651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:44.987577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46557 (local address 127.0.98.62:34409)
0927 02:42:44.987707 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:44.987711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:44.988513 (+ 802us) server_negotiation.cc:407] Connection header received
0927 02:42:44.989808 (+ 1295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:44.989811 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:44.989871 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:44.989943 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:44.991134 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:44.991670 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:44.993036 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:44.993313 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:44.993412 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:44.995848 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:44.995852 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:44.995854 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:44.995878 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:44.997140 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:44.997568 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:44.997570 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:44.997571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:44.997639 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:44.997927 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:44.997929 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:44.997930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:44.998087 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:44.998157 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:44.998686 (+ 529us) server_negotiation.cc:299] Negotiation successful
0927 02:42:44.998781 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 02:42:44.999676 20073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:44.987943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34409 (local address 127.0.98.3:46557)
0927 02:42:44.988363 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:44.988404 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:44.989599 (+ 1195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:44.990119 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:44.990127 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:44.990511 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:44.990987 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:44.991000 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:44.991780 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:44.991783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:44.992899 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:44.992909 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:44.993470 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:44.993475 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:44.993675 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:44.994285 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:42:44.994306 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:44.995708 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:44.997247 (+ 1539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:44.997252 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:44.997262 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:44.997493 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:44.997711 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:42:44.997713 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:42:44.997714 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:42:44.997841 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:42:44.998180 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:44.998185 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:44.998454 (+ 269us) client_negotiation.cc:769] Sending connection context
0927 02:42:44.998656 (+ 202us) client_negotiation.cc:240] Negotiation successful
0927 02:42:44.998901 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":113,"threads_started":1}
I20260927 02:42:45.000738 20071 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34409
I20260927 02:42:45.000954 20071 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:45.001405 20071 heartbeater.cc:511] Master 127.0.98.62:34409 requested a full tablet report, sending...
I20260927 02:42:45.002393 19611 ts_manager.cc:194] Registered new tserver with Master: 5334b3380b0f44c49e866ca1b6910fed (127.0.98.3:42261)
I20260927 02:42:45.002933 19611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46557
I20260927 02:42:45.005425 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:42:45.014132 19651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:45.007259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45168 (local address 127.0.98.62:34409)
0927 02:42:45.007387 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:45.007392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:45.007551 (+ 159us) server_negotiation.cc:407] Connection header received
0927 02:42:45.007673 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:45.007677 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:45.007718 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:45.007777 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:45.008405 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:45.008901 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:45.009690 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:45.009911 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:45.009995 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:45.011277 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:45.011281 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:45.011283 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:45.011308 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:45.012606 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:45.012993 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:45.012996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:45.012998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:45.013068 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:45.013306 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:45.013309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:45.013311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:45.013528 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:45.013645 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:45.013878 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 02:42:45.013982 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 02:42:45.017028 19611 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45168:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:42:45.020119 19611 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:42:45.035193 20087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:45.029756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59238 (local address 127.0.98.3:42261)
0927 02:42:45.030389 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:45.030401 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:45.030422 (+ 21us) server_negotiation.cc:407] Connection header received
0927 02:42:45.030496 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:45.030501 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:45.030735 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:45.031147 (+ 412us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:45.031288 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:45.032377 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:45.033743 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:45.034476 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:45.034614 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:45.034773 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:45.034931 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 02:42:45.035031 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":145,"threads_started":1}
I20260927 02:42:45.035192 20083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:45.028954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42261 (local address 127.0.0.1:59238)
0927 02:42:45.029867 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:45.029879 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:45.029952 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:45.030892 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:45.030895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:45.030913 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:45.031004 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:45.031010 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:45.032522 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:45.032525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:45.033633 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:45.033642 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:45.034624 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:45.034627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:45.034753 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:45.034798 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 02:42:45.034892 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 02:42:45.035035 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":854,"mutex_wait_us":32,"thread_start_us":82,"threads_started":1}
I20260927 02:42:45.035279 20084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:45.029085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58530 (local address 127.0.98.2:40747)
0927 02:42:45.029708 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:45.029721 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:45.029745 (+ 24us) server_negotiation.cc:407] Connection header received
0927 02:42:45.029972 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:45.029978 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:45.030224 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:45.030410 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:45.030905 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:45.031905 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:45.033450 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:45.034229 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:45.034348 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:45.034502 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:45.035011 (+ 509us) server_negotiation.cc:299] Negotiation successful
0927 02:42:45.035110 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":144,"threads_started":1}
I20260927 02:42:45.035859 20086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:45.029767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58344 (local address 127.0.98.1:37543)
0927 02:42:45.030243 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:45.030292 (+ 49us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:45.030593 (+ 301us) server_negotiation.cc:407] Connection header received
0927 02:42:45.030733 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:45.030738 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:45.030923 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:45.031068 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:45.031467 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:45.032656 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:45.034010 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:45.034843 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:45.034932 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:45.035055 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:45.035624 (+ 569us) server_negotiation.cc:299] Negotiation successful
0927 02:42:45.035707 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":117,"threads_started":1}
I20260927 02:42:45.036180 20085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:45.029148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37543 (local address 127.0.0.1:58344)
0927 02:42:45.030415 (+ 1267us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:45.030427 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:45.030508 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:45.031235 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:45.031238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:45.031279 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:45.031357 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:45.031363 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:45.032772 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:45.032775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:45.033884 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:45.033894 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:45.035110 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:45.035113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:45.035210 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:45.035231 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 02:42:45.035993 (+ 762us) client_negotiation.cc:240] Negotiation successful
0927 02:42:45.036066 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1209,"mutex_wait_us":63,"thread_start_us":93,"threads_started":1}
I20260927 02:42:45.036422 20082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:45.028942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40747 (local address 127.0.0.1:58530)
0927 02:42:45.029588 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:45.029641 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:45.029774 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:45.030613 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:45.030619 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:45.030641 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:45.030778 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:45.030784 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:45.032024 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:45.032027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:45.033316 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:45.033326 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:45.034565 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:45.034569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:45.034715 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:45.034783 (+ 68us) client_negotiation.cc:769] Sending connection context
0927 02:42:45.034899 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 02:42:45.036296 (+ 1397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"mutex_wait_us":1308,"thread_start_us":98,"threads_started":1}
I20260927 02:42:45.038128 20005 tablet_service.cc:1467] Processing CreateTablet for tablet f87f88a29bcb4dedbf0f15c5c95e6d19 (DEFAULT_TABLE table=test-table [id=aebfafe29cca429aa03b490d65be1e33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:45.039278 20005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f87f88a29bcb4dedbf0f15c5c95e6d19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:45.041098 19737 tablet_service.cc:1467] Processing CreateTablet for tablet f87f88a29bcb4dedbf0f15c5c95e6d19 (DEFAULT_TABLE table=test-table [id=aebfafe29cca429aa03b490d65be1e33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:45.041098 19871 tablet_service.cc:1467] Processing CreateTablet for tablet f87f88a29bcb4dedbf0f15c5c95e6d19 (DEFAULT_TABLE table=test-table [id=aebfafe29cca429aa03b490d65be1e33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:45.042162 19871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f87f88a29bcb4dedbf0f15c5c95e6d19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:45.042162 19737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f87f88a29bcb4dedbf0f15c5c95e6d19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:45.046022 20088 tablet_bootstrap.cc:492] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 5334b3380b0f44c49e866ca1b6910fed: Bootstrap starting.
I20260927 02:42:45.048193 20088 tablet_bootstrap.cc:654] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 5334b3380b0f44c49e866ca1b6910fed: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:45.049206 20089 tablet_bootstrap.cc:492] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 1ea2444bf029428cb6fd2c85d752b58e: Bootstrap starting.
I20260927 02:42:45.049377 20088 log.cc:824] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 5334b3380b0f44c49e866ca1b6910fed: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:45.049711 20090 tablet_bootstrap.cc:492] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 2251248973ce43b48f5119be4560d5cb: Bootstrap starting.
I20260927 02:42:45.051599 20088 tablet_bootstrap.cc:492] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 5334b3380b0f44c49e866ca1b6910fed: No bootstrap required, opened a new log
I20260927 02:42:45.051808 20088 ts_tablet_manager.cc:1397] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 5334b3380b0f44c49e866ca1b6910fed: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 02:42:45.051995 20090 tablet_bootstrap.cc:654] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 2251248973ce43b48f5119be4560d5cb: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:45.051995 20089 tablet_bootstrap.cc:654] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 1ea2444bf029428cb6fd2c85d752b58e: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:45.053211 20089 log.cc:824] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 1ea2444bf029428cb6fd2c85d752b58e: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:45.053212 20090 log.cc:824] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 2251248973ce43b48f5119be4560d5cb: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:45.055090 20090 tablet_bootstrap.cc:492] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 2251248973ce43b48f5119be4560d5cb: No bootstrap required, opened a new log
I20260927 02:42:45.054955 20088 raft_consensus.cc:348] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 5334b3380b0f44c49e866ca1b6910fed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea2444bf029428cb6fd2c85d752b58e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40747 } } peers { permanent_uuid: "5334b3380b0f44c49e866ca1b6910fed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42261 } } peers { permanent_uuid: "2251248973ce43b48f5119be4560d5cb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37543 } }
I20260927 02:42:45.055174 20088 raft_consensus.cc:374] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 5334b3380b0f44c49e866ca1b6910fed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:45.055240 20088 raft_consensus.cc:729] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 5334b3380b0f44c49e866ca1b6910fed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5334b3380b0f44c49e866ca1b6910fed, State: Initialized, Role: FOLLOWER
I20260927 02:42:45.055274 20090 ts_tablet_manager.cc:1397] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 2251248973ce43b48f5119be4560d5cb: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 02:42:45.055655 20088 consensus_queue.cc:260] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 5334b3380b0f44c49e866ca1b6910fed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea2444bf029428cb6fd2c85d752b58e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40747 } } peers { permanent_uuid: "5334b3380b0f44c49e866ca1b6910fed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42261 } } peers { permanent_uuid: "2251248973ce43b48f5119be4560d5cb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37543 } }
I20260927 02:42:45.056253 20071 heartbeater.cc:503] Master 127.0.98.62:34409 was elected leader, sending a full tablet report...
I20260927 02:42:45.056489 20088 ts_tablet_manager.cc:1428] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 5334b3380b0f44c49e866ca1b6910fed: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260927 02:42:45.057716 20089 tablet_bootstrap.cc:492] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 1ea2444bf029428cb6fd2c85d752b58e: No bootstrap required, opened a new log
I20260927 02:42:45.057907 20089 ts_tablet_manager.cc:1397] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 1ea2444bf029428cb6fd2c85d752b58e: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260927 02:42:45.059203 20090 raft_consensus.cc:348] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 2251248973ce43b48f5119be4560d5cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea2444bf029428cb6fd2c85d752b58e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40747 } } peers { permanent_uuid: "5334b3380b0f44c49e866ca1b6910fed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42261 } } peers { permanent_uuid: "2251248973ce43b48f5119be4560d5cb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37543 } }
I20260927 02:42:45.059475 20090 raft_consensus.cc:374] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 2251248973ce43b48f5119be4560d5cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:45.059522 20090 raft_consensus.cc:729] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 2251248973ce43b48f5119be4560d5cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2251248973ce43b48f5119be4560d5cb, State: Initialized, Role: FOLLOWER
I20260927 02:42:45.059981 20090 consensus_queue.cc:260] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 2251248973ce43b48f5119be4560d5cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea2444bf029428cb6fd2c85d752b58e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40747 } } peers { permanent_uuid: "5334b3380b0f44c49e866ca1b6910fed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42261 } } peers { permanent_uuid: "2251248973ce43b48f5119be4560d5cb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37543 } }
I20260927 02:42:45.060681 19803 heartbeater.cc:503] Master 127.0.98.62:34409 was elected leader, sending a full tablet report...
I20260927 02:42:45.061038 20090 ts_tablet_manager.cc:1428] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 2251248973ce43b48f5119be4560d5cb: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 02:42:45.061757 20089 raft_consensus.cc:348] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 1ea2444bf029428cb6fd2c85d752b58e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea2444bf029428cb6fd2c85d752b58e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40747 } } peers { permanent_uuid: "5334b3380b0f44c49e866ca1b6910fed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42261 } } peers { permanent_uuid: "2251248973ce43b48f5119be4560d5cb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37543 } }
I20260927 02:42:45.061993 20089 raft_consensus.cc:374] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 1ea2444bf029428cb6fd2c85d752b58e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:45.062055 20089 raft_consensus.cc:729] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 1ea2444bf029428cb6fd2c85d752b58e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ea2444bf029428cb6fd2c85d752b58e, State: Initialized, Role: FOLLOWER
I20260927 02:42:45.062474 20089 consensus_queue.cc:260] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 1ea2444bf029428cb6fd2c85d752b58e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea2444bf029428cb6fd2c85d752b58e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40747 } } peers { permanent_uuid: "5334b3380b0f44c49e866ca1b6910fed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42261 } } peers { permanent_uuid: "2251248973ce43b48f5119be4560d5cb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37543 } }
I20260927 02:42:45.063069 19937 heartbeater.cc:503] Master 127.0.98.62:34409 was elected leader, sending a full tablet report...
I20260927 02:42:45.063376 20089 ts_tablet_manager.cc:1428] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 1ea2444bf029428cb6fd2c85d752b58e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260927 02:42:45.070479 19804 tablet.cc:2369] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 2251248973ce43b48f5119be4560d5cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:42:45.125995 20094 raft_consensus.cc:482] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 5334b3380b0f44c49e866ca1b6910fed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:42:45.126199 20094 raft_consensus.cc:504] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 5334b3380b0f44c49e866ca1b6910fed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea2444bf029428cb6fd2c85d752b58e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40747 } } peers { permanent_uuid: "5334b3380b0f44c49e866ca1b6910fed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42261 } } peers { permanent_uuid: "2251248973ce43b48f5119be4560d5cb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37543 } }
I20260927 02:42:45.127419 20094 leader_election.cc:288] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 5334b3380b0f44c49e866ca1b6910fed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ea2444bf029428cb6fd2c85d752b58e (127.0.98.2:40747), 2251248973ce43b48f5119be4560d5cb (127.0.98.1:37543)
I20260927 02:42:45.130954 20086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:45.127797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58021 (local address 127.0.98.1:37543)
0927 02:42:45.127920 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:45.127925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:45.127940 (+ 15us) server_negotiation.cc:407] Connection header received
0927 02:42:45.128014 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:45.128017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:45.128062 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:45.128138 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:45.128515 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:45.129014 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:45.130029 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:45.130458 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:45.130550 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:45.130611 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:45.130779 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 02:42:45.130849 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 02:42:45.130954 20073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:45.127657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37543 (local address 127.0.98.3:58021)
0927 02:42:45.127766 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:45.127782 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:45.127890 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:45.128261 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:45.128264 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:45.128283 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:45.128405 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:45.128412 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:45.129124 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:45.129128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:45.129910 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:45.129918 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:45.130569 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:45.130573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:45.130676 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:45.130694 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 02:42:45.130758 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 02:42:45.130826 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 02:42:45.131201 20097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:45.127658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40747 (local address 127.0.98.3:60005)
0927 02:42:45.128244 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:45.128256 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:45.128340 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:45.128560 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:45.128563 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:45.128581 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:45.128691 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:45.128696 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:45.129371 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:45.129375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:45.130151 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:45.130159 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:45.130783 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:45.130786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:45.130888 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:45.130929 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 02:42:45.130997 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 02:42:45.131062 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"mutex_wait_us":299,"thread_start_us":109,"threads_started":1}
I20260927 02:42:45.131201 20084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:45.127803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60005 (local address 127.0.98.2:40747)
0927 02:42:45.127921 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:45.127925 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:45.128342 (+ 417us) server_negotiation.cc:407] Connection header received
0927 02:42:45.128418 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:45.128420 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:45.128461 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:45.128526 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:45.128799 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:45.129276 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:45.130290 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:45.130682 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:45.130770 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:45.130824 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:45.131024 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 02:42:45.131090 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 02:42:45.131635 19757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f87f88a29bcb4dedbf0f15c5c95e6d19" candidate_uuid: "5334b3380b0f44c49e866ca1b6910fed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2251248973ce43b48f5119be4560d5cb" is_pre_election: true
I20260927 02:42:45.131855 19891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f87f88a29bcb4dedbf0f15c5c95e6d19" candidate_uuid: "5334b3380b0f44c49e866ca1b6910fed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ea2444bf029428cb6fd2c85d752b58e" is_pre_election: true
I20260927 02:42:45.131978 19757 raft_consensus.cc:2454] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 2251248973ce43b48f5119be4560d5cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5334b3380b0f44c49e866ca1b6910fed in term 0.
I20260927 02:42:45.132117 19891 raft_consensus.cc:2454] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 1ea2444bf029428cb6fd2c85d752b58e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5334b3380b0f44c49e866ca1b6910fed in term 0.
I20260927 02:42:45.132467 19960 leader_election.cc:302] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 5334b3380b0f44c49e866ca1b6910fed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1ea2444bf029428cb6fd2c85d752b58e, 2251248973ce43b48f5119be4560d5cb, 5334b3380b0f44c49e866ca1b6910fed; no voters:
I20260927 02:42:45.132714 20094 raft_consensus.cc:2781] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 5334b3380b0f44c49e866ca1b6910fed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:42:45.132787 20094 raft_consensus.cc:482] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 5334b3380b0f44c49e866ca1b6910fed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:42:45.132817 20094 raft_consensus.cc:3037] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 5334b3380b0f44c49e866ca1b6910fed [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:45.133846 20094 raft_consensus.cc:504] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 5334b3380b0f44c49e866ca1b6910fed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea2444bf029428cb6fd2c85d752b58e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40747 } } peers { permanent_uuid: "5334b3380b0f44c49e866ca1b6910fed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42261 } } peers { permanent_uuid: "2251248973ce43b48f5119be4560d5cb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37543 } }
I20260927 02:42:45.134157 20094 leader_election.cc:288] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 5334b3380b0f44c49e866ca1b6910fed [CANDIDATE]: Term 1 election: Requested vote from peers 1ea2444bf029428cb6fd2c85d752b58e (127.0.98.2:40747), 2251248973ce43b48f5119be4560d5cb (127.0.98.1:37543)
I20260927 02:42:45.134505 19757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f87f88a29bcb4dedbf0f15c5c95e6d19" candidate_uuid: "5334b3380b0f44c49e866ca1b6910fed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2251248973ce43b48f5119be4560d5cb"
I20260927 02:42:45.134505 19891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f87f88a29bcb4dedbf0f15c5c95e6d19" candidate_uuid: "5334b3380b0f44c49e866ca1b6910fed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ea2444bf029428cb6fd2c85d752b58e"
I20260927 02:42:45.134636 19757 raft_consensus.cc:3037] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 2251248973ce43b48f5119be4560d5cb [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:45.134636 19891 raft_consensus.cc:3037] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 1ea2444bf029428cb6fd2c85d752b58e [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:45.135828 19891 raft_consensus.cc:2454] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 1ea2444bf029428cb6fd2c85d752b58e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5334b3380b0f44c49e866ca1b6910fed in term 1.
I20260927 02:42:45.135828 19757 raft_consensus.cc:2454] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 2251248973ce43b48f5119be4560d5cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5334b3380b0f44c49e866ca1b6910fed in term 1.
I20260927 02:42:45.136140 19961 leader_election.cc:302] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 5334b3380b0f44c49e866ca1b6910fed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1ea2444bf029428cb6fd2c85d752b58e, 2251248973ce43b48f5119be4560d5cb, 5334b3380b0f44c49e866ca1b6910fed; no voters:
I20260927 02:42:45.136358 20094 raft_consensus.cc:2781] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 5334b3380b0f44c49e866ca1b6910fed [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:45.136579 20094 raft_consensus.cc:686] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 5334b3380b0f44c49e866ca1b6910fed [term 1 LEADER]: Becoming Leader. State: Replica: 5334b3380b0f44c49e866ca1b6910fed, State: Running, Role: LEADER
I20260927 02:42:45.136907 20094 consensus_queue.cc:237] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 5334b3380b0f44c49e866ca1b6910fed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea2444bf029428cb6fd2c85d752b58e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40747 } } peers { permanent_uuid: "5334b3380b0f44c49e866ca1b6910fed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42261 } } peers { permanent_uuid: "2251248973ce43b48f5119be4560d5cb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37543 } }
I20260927 02:42:45.140038 19611 catalog_manager.cc:5495] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 5334b3380b0f44c49e866ca1b6910fed reported cstate change: term changed from 0 to 1, leader changed from <none> to 5334b3380b0f44c49e866ca1b6910fed (127.0.98.3). New cstate: current_term: 1 leader_uuid: "5334b3380b0f44c49e866ca1b6910fed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea2444bf029428cb6fd2c85d752b58e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5334b3380b0f44c49e866ca1b6910fed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2251248973ce43b48f5119be4560d5cb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37543 } health_report { overall_health: UNKNOWN } } }
I20260927 02:42:45.156762 20087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:45.153303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59254 (local address 127.0.98.3:42261)
0927 02:42:45.153428 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:45.153431 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:45.153444 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:42:45.153584 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:45.153586 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:45.153639 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:45.153719 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:42:45.153912 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:45.154410 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:45.155315 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:45.155530 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:45.155684 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:45.155939 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:42:45.156350 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:42:45.156454 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:45.156481 (+ 27us) server_negotiation.cc:299] Negotiation successful
0927 02:42:45.156530 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 02:42:45.162273 19891 raft_consensus.cc:1261] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 1ea2444bf029428cb6fd2c85d752b58e [term 1 FOLLOWER]: Refusing update from remote peer 5334b3380b0f44c49e866ca1b6910fed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:42:45.162274 19757 raft_consensus.cc:1261] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 2251248973ce43b48f5119be4560d5cb [term 1 FOLLOWER]: Refusing update from remote peer 5334b3380b0f44c49e866ca1b6910fed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:42:45.162923 20094 consensus_queue.cc:1035] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 5334b3380b0f44c49e866ca1b6910fed [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ea2444bf029428cb6fd2c85d752b58e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:42:45.163113 20098 consensus_queue.cc:1035] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 5334b3380b0f44c49e866ca1b6910fed [LEADER]: Connected to new peer: Peer: permanent_uuid: "2251248973ce43b48f5119be4560d5cb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:42:45.171763 20103 mvcc.cc:205] Tried to move back new op lower bound from 7333793649297944576 to 7333793649203421184. Current Snapshot: MvccSnapshot[applied={T|T < 7333793649297944576}]
I20260927 02:42:45.178056 19611 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:42:45.178205 19611 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:42:45.180599 19611 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b8d6cd4094394687ae1d8f49e733e160: Sending DeleteTablet for 3 replicas of tablet f87f88a29bcb4dedbf0f15c5c95e6d19
I20260927 02:42:45.181301 20005 tablet_service.cc:1514] Processing DeleteTablet for tablet f87f88a29bcb4dedbf0f15c5c95e6d19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59238
I20260927 02:42:45.181315 19737 tablet_service.cc:1514] Processing DeleteTablet for tablet f87f88a29bcb4dedbf0f15c5c95e6d19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58344
I20260927 02:42:45.181793 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 19674
I20260927 02:42:45.181813 20111 tablet_replica.cc:330] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 5334b3380b0f44c49e866ca1b6910fed: stopping tablet replica
I20260927 02:42:45.181967 19871 tablet_service.cc:1514] Processing DeleteTablet for tablet f87f88a29bcb4dedbf0f15c5c95e6d19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58530
I20260927 02:42:45.182490 20111 raft_consensus.cc:2229] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 5334b3380b0f44c49e866ca1b6910fed [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:42:45.182853 20111 raft_consensus.cc:2258] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 5334b3380b0f44c49e866ca1b6910fed [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 02:42:45.187561 19598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37543 (error 108)
I20260927 02:42:45.187655 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 19808
I20260927 02:42:45.188793 20111 ts_tablet_manager.cc:1905] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 5334b3380b0f44c49e866ca1b6910fed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:42:45.188860 20085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:45.188553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37543 (local address 127.0.0.1:58358)
0927 02:42:45.188718 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:45.188780 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260927 02:42:45.189103 19598 catalog_manager.cc:4584] TS 2251248973ce43b48f5119be4560d5cb (127.0.98.1:37543): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f87f88a29bcb4dedbf0f15c5c95e6d19: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37543: connect: Connection refused (error 111)
I20260927 02:42:45.191301 20111 ts_tablet_manager.cc:1918] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 5334b3380b0f44c49e866ca1b6910fed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:42:45.191387 20111 log.cc:1196] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 5334b3380b0f44c49e866ca1b6910fed: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790476718023131-392-0/minicluster-data/ts-2/wal/wals/f87f88a29bcb4dedbf0f15c5c95e6d19
I20260927 02:42:45.191642 20111 ts_tablet_manager.cc:1939] T f87f88a29bcb4dedbf0f15c5c95e6d19 P 5334b3380b0f44c49e866ca1b6910fed: Deleting consensus metadata
I20260927 02:42:45.194661 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 19942
I20260927 02:42:45.195340 20085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:45.194797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40747 (local address 127.0.0.1:58532)
0927 02:42:45.195211 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:45.195264 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:40747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341}
I20260927 02:42:45.199556 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 19580
2026-09-27T02:42:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2135 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T02:42:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:42:45Z Disabled control of system clock
I20260927 02:42:45.226964 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43957
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 02:42:45.334163 20117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:45.334444 20117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:45.334479 20117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:45.338486 20117 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:42:45.338542 20117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:45.338557 20117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:45.338577 20117 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:42:45.338591 20117 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:42:45.343215 20117 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43957
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:42:45.344172 20117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:45.345439 20117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:45.353664 20123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:45.353667 20122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:45.353794 20125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:45.353870 20117 server_base.cc:1034] running on GCE node
I20260927 02:42:45.354606 20117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:45.355723 20117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:45.356894 20117 hybrid_clock.cc:648] HybridClock initialized: now 1790476965356878 us; error 29 us; skew 500 ppm
I20260927 02:42:45.358798 20117 webserver.cc:460] Webserver started at http://127.0.98.62:40083/ using document root <none> and password file <none>
I20260927 02:42:45.359356 20117 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:45.359421 20117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:45.359601 20117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:45.361611 20117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/master-0/data/instance:
uuid: "da91707e150e4e17973d356007f8d8c3"
format_stamp: "Formatted at 2026-09-27 02:42:45 on dist-test-slave-zv8z"
server_key: "2a99ede096cd5c935fc95974cddbfafa"
server_key_iv: "7cf23c34ba94543c61c092f4c00182a3"
server_key_version: "encryptionkey@0"
I20260927 02:42:45.362128 20117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/master-0/wal/instance:
uuid: "da91707e150e4e17973d356007f8d8c3"
format_stamp: "Formatted at 2026-09-27 02:42:45 on dist-test-slave-zv8z"
server_key: "2a99ede096cd5c935fc95974cddbfafa"
server_key_iv: "7cf23c34ba94543c61c092f4c00182a3"
server_key_version: "encryptionkey@0"
I20260927 02:42:45.365562 20117 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 02:42:45.367851 20131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:45.369010 20117 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:42:45.369117 20117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "da91707e150e4e17973d356007f8d8c3"
format_stamp: "Formatted at 2026-09-27 02:42:45 on dist-test-slave-zv8z"
server_key: "2a99ede096cd5c935fc95974cddbfafa"
server_key_iv: "7cf23c34ba94543c61c092f4c00182a3"
server_key_version: "encryptionkey@0"
I20260927 02:42:45.369206 20117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:45.388087 20117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:45.388784 20117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:45.388960 20117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:45.395498 20117 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43957
I20260927 02:42:45.395553 20183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43957 every 8 connection(s)
I20260927 02:42:45.396418 20117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:42:45.399529 20184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:45.402042 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 20117
I20260927 02:42:45.402184 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/master-0/wal/instance
I20260927 02:42:45.402473 392 external_mini_cluster.cc:1428] Setting key 00b3c7cabce776b975e3735ee7f1d0d0
I20260927 02:42:45.404949 20184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da91707e150e4e17973d356007f8d8c3: Bootstrap starting.
I20260927 02:42:45.407142 20184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da91707e150e4e17973d356007f8d8c3: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:45.407974 20184 log.cc:824] T 00000000000000000000000000000000 P da91707e150e4e17973d356007f8d8c3: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:45.409902 20184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da91707e150e4e17973d356007f8d8c3: No bootstrap required, opened a new log
I20260927 02:42:45.411326 20187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:45.403843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36968 (local address 127.0.98.62:43957)
0927 02:42:45.404442 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:45.404457 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:45.404492 (+ 35us) server_negotiation.cc:407] Connection header received
0927 02:42:45.405131 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:45.405149 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:45.405536 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:45.405808 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:45.406134 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:45.407123 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:45.408278 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:45.408729 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:45.408870 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:45.409646 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:45.409651 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:45.409663 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:42:45.410063 (+ 400us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:45.410118 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:42:45.410139 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:45.410210 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:45.410378 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 02:42:45.410629 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":113,"threads_started":1}
I20260927 02:42:45.413635 20184 raft_consensus.cc:348] T 00000000000000000000000000000000 P da91707e150e4e17973d356007f8d8c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da91707e150e4e17973d356007f8d8c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43957 } }
I20260927 02:42:45.413843 20184 raft_consensus.cc:374] T 00000000000000000000000000000000 P da91707e150e4e17973d356007f8d8c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:45.413897 20184 raft_consensus.cc:729] T 00000000000000000000000000000000 P da91707e150e4e17973d356007f8d8c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da91707e150e4e17973d356007f8d8c3, State: Initialized, Role: FOLLOWER
I20260927 02:42:45.414310 20184 consensus_queue.cc:260] T 00000000000000000000000000000000 P da91707e150e4e17973d356007f8d8c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da91707e150e4e17973d356007f8d8c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43957 } }
I20260927 02:42:45.414427 20184 raft_consensus.cc:388] T 00000000000000000000000000000000 P da91707e150e4e17973d356007f8d8c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:42:45.414484 20184 raft_consensus.cc:482] T 00000000000000000000000000000000 P da91707e150e4e17973d356007f8d8c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:42:45.414589 20184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da91707e150e4e17973d356007f8d8c3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:45.415374 20184 raft_consensus.cc:504] T 00000000000000000000000000000000 P da91707e150e4e17973d356007f8d8c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da91707e150e4e17973d356007f8d8c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43957 } }
I20260927 02:42:45.415676 20184 leader_election.cc:302] T 00000000000000000000000000000000 P da91707e150e4e17973d356007f8d8c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da91707e150e4e17973d356007f8d8c3; no voters:
I20260927 02:42:45.415925 20184 leader_election.cc:288] T 00000000000000000000000000000000 P da91707e150e4e17973d356007f8d8c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:42:45.416021 20189 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da91707e150e4e17973d356007f8d8c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:45.416225 20189 raft_consensus.cc:686] T 00000000000000000000000000000000 P da91707e150e4e17973d356007f8d8c3 [term 1 LEADER]: Becoming Leader. State: Replica: da91707e150e4e17973d356007f8d8c3, State: Running, Role: LEADER
I20260927 02:42:45.416514 20189 consensus_queue.cc:237] T 00000000000000000000000000000000 P da91707e150e4e17973d356007f8d8c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da91707e150e4e17973d356007f8d8c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43957 } }
I20260927 02:42:45.416957 20184 sys_catalog.cc:564] T 00000000000000000000000000000000 P da91707e150e4e17973d356007f8d8c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:42:45.417917 20190 sys_catalog.cc:455] T 00000000000000000000000000000000 P da91707e150e4e17973d356007f8d8c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da91707e150e4e17973d356007f8d8c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da91707e150e4e17973d356007f8d8c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43957 } } }
I20260927 02:42:45.418030 20190 sys_catalog.cc:458] T 00000000000000000000000000000000 P da91707e150e4e17973d356007f8d8c3 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:45.418326 20191 sys_catalog.cc:455] T 00000000000000000000000000000000 P da91707e150e4e17973d356007f8d8c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da91707e150e4e17973d356007f8d8c3. Latest consensus state: current_term: 1 leader_uuid: "da91707e150e4e17973d356007f8d8c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da91707e150e4e17973d356007f8d8c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43957 } } }
I20260927 02:42:45.418417 20191 sys_catalog.cc:458] T 00000000000000000000000000000000 P da91707e150e4e17973d356007f8d8c3 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:45.418490 20198 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:42:45.421602 20198 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:42:45.426769 20198 catalog_manager.cc:1347] Generated new cluster ID: f2a3579c3b3d458bb9160f4e5276d9e7
I20260927 02:42:45.426839 20198 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:42:45.435309 20198 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:42:45.436134 20198 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:42:45.446483 20198 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da91707e150e4e17973d356007f8d8c3: Generated new TSK 0
I20260927 02:42:45.447104 20198 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:42:45.452468 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43957
--builtin_ntp_servers=127.0.98.20:38955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 02:42:45.558917 20208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:45.559155 20208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:45.559204 20208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:45.563138 20208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:45.563190 20208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:45.563263 20208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:42:45.567934 20208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:42:45.568946 20208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:45.570269 20208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:45.579421 20214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:45.579391 20216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:45.579396 20213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:45.579770 20208 server_base.cc:1034] running on GCE node
I20260927 02:42:45.580086 20208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:45.580636 20208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:45.581811 20208 hybrid_clock.cc:648] HybridClock initialized: now 1790476965581796 us; error 29 us; skew 500 ppm
I20260927 02:42:45.583566 20208 webserver.cc:460] Webserver started at http://127.0.98.1:33633/ using document root <none> and password file <none>
I20260927 02:42:45.584069 20208 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:45.584128 20208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:45.584290 20208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:45.586184 20208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/ts-0/data/instance:
uuid: "67a8864df8fa45a2bd043ba18d6be889"
format_stamp: "Formatted at 2026-09-27 02:42:45 on dist-test-slave-zv8z"
server_key: "dcb803d3728a36a48e6ddf4940ce0f25"
server_key_iv: "c07324929a614d93dee55bb05bc8fab1"
server_key_version: "encryptionkey@0"
I20260927 02:42:45.586723 20208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/ts-0/wal/instance:
uuid: "67a8864df8fa45a2bd043ba18d6be889"
format_stamp: "Formatted at 2026-09-27 02:42:45 on dist-test-slave-zv8z"
server_key: "dcb803d3728a36a48e6ddf4940ce0f25"
server_key_iv: "c07324929a614d93dee55bb05bc8fab1"
server_key_version: "encryptionkey@0"
I20260927 02:42:45.590229 20208 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 02:42:45.592550 20222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:45.593775 20208 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:42:45.593869 20208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/ts-0/wal
uuid: "67a8864df8fa45a2bd043ba18d6be889"
format_stamp: "Formatted at 2026-09-27 02:42:45 on dist-test-slave-zv8z"
server_key: "dcb803d3728a36a48e6ddf4940ce0f25"
server_key_iv: "c07324929a614d93dee55bb05bc8fab1"
server_key_version: "encryptionkey@0"
I20260927 02:42:45.593976 20208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:45.611166 20208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:45.611794 20208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:45.611951 20208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:45.612491 20208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:45.613351 20208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:45.613396 20208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:45.613435 20208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:45.613451 20208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:45.621810 20208 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33411
I20260927 02:42:45.621848 20335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33411 every 8 connection(s)
I20260927 02:42:45.622680 20208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:42:45.628613 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 20208
I20260927 02:42:45.628795 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:42:45.629107 392 external_mini_cluster.cc:1428] Setting key f69229f958a01c8ea447f5636ae4250f
I20260927 02:42:45.631088 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43957
--builtin_ntp_servers=127.0.98.20:38955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 02:42:45.632136 20187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:45.624348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38819 (local address 127.0.98.62:43957)
0927 02:42:45.624470 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:45.624473 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:45.625655 (+ 1182us) server_negotiation.cc:407] Connection header received
0927 02:42:45.626951 (+ 1296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:45.626954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:45.627002 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:45.627078 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:45.627970 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:45.628534 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:45.629970 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:45.630287 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:45.630381 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:45.631698 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:45.631701 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:45.631703 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:42:45.631714 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:45.631733 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:42:45.631741 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:45.631801 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:45.631974 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 02:42:45.632025 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":40}
I20260927 02:42:45.632978 20338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:45.624701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43957 (local address 127.0.98.1:38819)
0927 02:42:45.625471 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:45.625512 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:45.626761 (+ 1249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:45.627241 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:45.627248 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:45.627329 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:45.627812 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:45.627825 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:45.628707 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:45.628711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:45.629844 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:45.629855 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:45.630593 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:45.630597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:45.630781 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:45.631401 (+ 620us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:42:45.631403 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:45.631455 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:42:45.631533 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:45.631815 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:45.631822 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:45.631827 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:42:45.631935 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 02:42:45.632212 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":114,"threads_started":1}
I20260927 02:42:45.634132 20336 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43957
I20260927 02:42:45.634354 20336 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:45.634892 20336 heartbeater.cc:511] Master 127.0.98.62:43957 requested a full tablet report, sending...
I20260927 02:42:45.636291 20148 ts_manager.cc:194] Registered new tserver with Master: 67a8864df8fa45a2bd043ba18d6be889 (127.0.98.1:33411)
I20260927 02:42:45.637710 20148 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:38819
W20260927 02:42:45.738898 20339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:45.739161 20339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:45.739214 20339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:45.743127 20339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:45.743173 20339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:45.743247 20339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:42:45.748201 20339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:42:45.749233 20339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:45.750625 20339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:45.760208 20345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:45.760221 20339 server_base.cc:1034] running on GCE node
W20260927 02:42:45.760303 20344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:45.760460 20347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:45.760955 20339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:45.761555 20339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:45.762737 20339 hybrid_clock.cc:648] HybridClock initialized: now 1790476965762719 us; error 34 us; skew 500 ppm
I20260927 02:42:45.764467 20339 webserver.cc:460] Webserver started at http://127.0.98.2:40139/ using document root <none> and password file <none>
I20260927 02:42:45.764995 20339 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:45.765062 20339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:45.765240 20339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:45.767318 20339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/ts-1/data/instance:
uuid: "b3de8d38e8474754b3dc3c3d32f53ad3"
format_stamp: "Formatted at 2026-09-27 02:42:45 on dist-test-slave-zv8z"
server_key: "1f3b9c353ca357a8830b4346cd2a6ed2"
server_key_iv: "83a85c633d2bd09537306aa612037b11"
server_key_version: "encryptionkey@0"
I20260927 02:42:45.767838 20339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b3de8d38e8474754b3dc3c3d32f53ad3"
format_stamp: "Formatted at 2026-09-27 02:42:45 on dist-test-slave-zv8z"
server_key: "1f3b9c353ca357a8830b4346cd2a6ed2"
server_key_iv: "83a85c633d2bd09537306aa612037b11"
server_key_version: "encryptionkey@0"
I20260927 02:42:45.771313 20339 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:42:45.773705 20353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:45.774983 20339 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:42:45.775085 20339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/ts-1/wal
uuid: "b3de8d38e8474754b3dc3c3d32f53ad3"
format_stamp: "Formatted at 2026-09-27 02:42:45 on dist-test-slave-zv8z"
server_key: "1f3b9c353ca357a8830b4346cd2a6ed2"
server_key_iv: "83a85c633d2bd09537306aa612037b11"
server_key_version: "encryptionkey@0"
I20260927 02:42:45.775171 20339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:45.793877 20339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:45.794534 20339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:45.794709 20339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:45.795243 20339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:45.796118 20339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:45.796162 20339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:45.796211 20339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:45.796237 20339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:45.804147 20339 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41637
I20260927 02:42:45.804171 20466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41637 every 8 connection(s)
I20260927 02:42:45.804972 20339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:42:45.807216 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 20339
I20260927 02:42:45.807349 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:42:45.807704 392 external_mini_cluster.cc:1428] Setting key 3511b61f16897d82a921696ce70044f8
I20260927 02:42:45.809733 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43957
--builtin_ntp_servers=127.0.98.20:38955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 02:42:45.814752 20187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:45.806794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38953 (local address 127.0.98.62:43957)
0927 02:42:45.806913 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:45.806916 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:45.808150 (+ 1234us) server_negotiation.cc:407] Connection header received
0927 02:42:45.809566 (+ 1416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:45.809569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:45.809623 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:45.809714 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:45.810693 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:45.811301 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:45.812751 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:45.812985 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:45.813083 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:45.814239 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:45.814242 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:45.814244 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:42:45.814278 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:45.814301 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:42:45.814311 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:45.814371 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:45.814535 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 02:42:45.814645 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 02:42:45.815861 20469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:45.807199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43957 (local address 127.0.98.2:38953)
0927 02:42:45.807946 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:45.807988 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:45.809367 (+ 1379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:45.809928 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:45.809938 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:45.810026 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:45.810558 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:45.810573 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:45.811424 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:45.811426 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:45.812600 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:45.812611 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:45.813137 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:45.813141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:45.813342 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:45.813938 (+ 596us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:42:45.813940 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:45.813996 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:42:45.814080 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:45.814387 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:45.814396 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:45.814402 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 02:42:45.814538 (+ 136us) client_negotiation.cc:240] Negotiation successful
0927 02:42:45.814851 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":138,"threads_started":1}
I20260927 02:42:45.817016 20467 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43957
I20260927 02:42:45.817245 20467 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:45.817770 20467 heartbeater.cc:511] Master 127.0.98.62:43957 requested a full tablet report, sending...
I20260927 02:42:45.818845 20148 ts_manager.cc:194] Registered new tserver with Master: b3de8d38e8474754b3dc3c3d32f53ad3 (127.0.98.2:41637)
I20260927 02:42:45.819437 20148 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:38953
W20260927 02:42:45.916400 20470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:45.916657 20470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:45.916708 20470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:45.921068 20470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:45.921139 20470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:45.921216 20470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:42:45.926020 20470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:42:45.927058 20470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:45.928391 20470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:45.937505 20476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:45.937512 20478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:45.937757 20470 server_base.cc:1034] running on GCE node
W20260927 02:42:45.937868 20475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:45.938311 20470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:45.938958 20470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:45.940161 20470 hybrid_clock.cc:648] HybridClock initialized: now 1790476965940148 us; error 40 us; skew 500 ppm
I20260927 02:42:45.941973 20470 webserver.cc:460] Webserver started at http://127.0.98.3:43473/ using document root <none> and password file <none>
I20260927 02:42:45.942571 20470 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:45.942641 20470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:45.942823 20470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:45.944832 20470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/ts-2/data/instance:
uuid: "77547a85f3d94d139b01967920d1ea72"
format_stamp: "Formatted at 2026-09-27 02:42:45 on dist-test-slave-zv8z"
server_key: "e7cf385b73d45beef80f46690b5b301a"
server_key_iv: "b875a20ec86cef5f497410ddefe3ab21"
server_key_version: "encryptionkey@0"
I20260927 02:42:45.945389 20470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/ts-2/wal/instance:
uuid: "77547a85f3d94d139b01967920d1ea72"
format_stamp: "Formatted at 2026-09-27 02:42:45 on dist-test-slave-zv8z"
server_key: "e7cf385b73d45beef80f46690b5b301a"
server_key_iv: "b875a20ec86cef5f497410ddefe3ab21"
server_key_version: "encryptionkey@0"
I20260927 02:42:45.948959 20470 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260927 02:42:45.951408 20484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:45.952631 20470 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:42:45.952742 20470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/ts-2/wal
uuid: "77547a85f3d94d139b01967920d1ea72"
format_stamp: "Formatted at 2026-09-27 02:42:45 on dist-test-slave-zv8z"
server_key: "e7cf385b73d45beef80f46690b5b301a"
server_key_iv: "b875a20ec86cef5f497410ddefe3ab21"
server_key_version: "encryptionkey@0"
I20260927 02:42:45.952838 20470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:45.969816 20470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:45.970522 20470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:45.970688 20470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:45.971252 20470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:42:45.972220 20470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:42:45.972266 20470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:45.972317 20470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:42:45.972345 20470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:45.980667 20470 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39525
I20260927 02:42:45.980865 20597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39525 every 8 connection(s)
I20260927 02:42:45.981611 20470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:42:45.985775 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 20470
I20260927 02:42:45.985881 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:42:45.986186 392 external_mini_cluster.cc:1428] Setting key cde5127159fe71c4d2256c4321711a30
I20260927 02:42:45.990993 20187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:45.983490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56837 (local address 127.0.98.62:43957)
0927 02:42:45.983639 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:45.983644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:45.984738 (+ 1094us) server_negotiation.cc:407] Connection header received
0927 02:42:45.986108 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:45.986112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:45.986171 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:45.986282 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:45.987342 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:45.988057 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:45.989169 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:45.989391 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:45.989476 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:45.990488 (+ 1012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:45.990492 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:45.990495 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:42:45.990508 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:45.990528 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:42:45.990536 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:45.990653 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:45.990799 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 02:42:45.990856 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 02:42:45.991822 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:45.983900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43957 (local address 127.0.98.3:56837)
0927 02:42:45.984584 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:45.984631 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:45.985849 (+ 1218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:45.986463 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:45.986470 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:45.986558 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:42:45.987205 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:45.987223 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:45.988176 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:45.988179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:45.989040 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:45.989049 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:45.989525 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:45.989529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:45.989687 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:45.990215 (+ 528us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:42:45.990217 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:42:45.990286 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:42:45.990352 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:42:45.990662 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:42:45.990671 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:42:45.990677 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 02:42:45.990787 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 02:42:45.991060 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":133,"threads_started":1}
I20260927 02:42:45.992781 20598 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43957
I20260927 02:42:45.992991 20598 heartbeater.cc:464] Registering TS with master...
I20260927 02:42:45.993474 20598 heartbeater.cc:511] Master 127.0.98.62:43957 requested a full tablet report, sending...
I20260927 02:42:45.994503 20148 ts_manager.cc:194] Registered new tserver with Master: 77547a85f3d94d139b01967920d1ea72 (127.0.98.3:39525)
I20260927 02:42:45.995057 20148 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:56837
I20260927 02:42:45.999944 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:42:46.005477 20187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:46.001799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36980 (local address 127.0.98.62:43957)
0927 02:42:46.001902 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:46.001905 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:46.002023 (+ 118us) server_negotiation.cc:407] Connection header received
0927 02:42:46.002172 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:46.002174 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:46.002215 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:46.002323 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:46.002587 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:46.003093 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:46.003996 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:46.004218 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:46.004297 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:46.005064 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:46.005067 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:46.005069 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:42:46.005102 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:46.005122 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:42:46.005130 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:46.005187 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:46.005318 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 02:42:46.005372 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 02:42:46.008297 20148 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36980:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:42:46.011219 20148 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:42:46.024735 20613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:46.019820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46112 (local address 127.0.98.2:41637)
0927 02:42:46.020266 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:46.020279 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:46.020304 (+ 25us) server_negotiation.cc:407] Connection header received
0927 02:42:46.020380 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:46.020386 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:46.020582 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:46.020723 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:46.021004 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:46.021801 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:46.023329 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:46.024088 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:46.024185 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:46.024374 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:46.024476 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 02:42:46.024572 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":102,"threads_started":1}
I20260927 02:42:46.024724 20611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:46.019313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41637 (local address 127.0.0.1:46112)
0927 02:42:46.020050 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:46.020064 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:46.020153 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:46.020729 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:46.020734 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:46.020755 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:46.020883 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:46.020891 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:46.021917 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:46.021921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:46.023192 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:46.023203 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:46.024198 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:46.024202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:46.024302 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:46.024338 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 02:42:46.024446 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 02:42:46.024581 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"mutex_wait_us":27,"thread_start_us":106,"threads_started":1}
I20260927 02:42:46.025069 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:46.019170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42520 (local address 127.0.98.1:33411)
0927 02:42:46.020415 (+ 1245us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:46.020427 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:46.020450 (+ 23us) server_negotiation.cc:407] Connection header received
0927 02:42:46.020528 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:46.020533 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:46.020766 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:46.020913 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:46.021095 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:46.022283 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:46.023536 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:46.024189 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:46.024582 (+ 393us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:46.024767 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:46.024817 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 02:42:46.024911 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1139,"thread_start_us":129,"threads_started":1}
I20260927 02:42:46.025259 20609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:46.019008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33411 (local address 127.0.0.1:42520)
0927 02:42:46.019354 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:46.019406 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:46.019521 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:46.020909 (+ 1388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:46.020912 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:46.020928 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:46.021011 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:46.021017 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:46.022396 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:46.022399 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:46.023426 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:46.023433 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:46.024322 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:46.024326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:46.024410 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:46.024435 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 02:42:46.024482 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 02:42:46.025126 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":558,"thread_start_us":105,"threads_started":1}
I20260927 02:42:46.026461 20614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:46.019919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53966 (local address 127.0.98.3:39525)
0927 02:42:46.020478 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:46.020500 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:46.020517 (+ 17us) server_negotiation.cc:407] Connection header received
0927 02:42:46.020575 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:46.020579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:46.020766 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:46.020914 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:46.021293 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:46.022513 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:46.023876 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:46.024590 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:46.024680 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:46.024814 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:46.025406 (+ 592us) server_negotiation.cc:299] Negotiation successful
0927 02:42:46.026344 (+ 938us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":157,"threads_started":1}
I20260927 02:42:46.025309 20612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:46.019771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39525 (local address 127.0.0.1:53966)
0927 02:42:46.020262 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:46.020275 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:46.020360 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:46.021047 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:46.021050 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:46.021069 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:46.021169 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:46.021175 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:46.022621 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:46.022624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:46.023753 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:46.023764 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:46.024930 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:46.024933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:46.025023 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:46.025043 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 02:42:46.025117 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 02:42:46.025187 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"mutex_wait_us":24,"thread_start_us":88,"threads_started":1}
I20260927 02:42:46.028350 20270 tablet_service.cc:1467] Processing CreateTablet for tablet 7afa0eba34a3419b85cd94a0fbe2f7c7 (DEFAULT_TABLE table=test-table [id=3296325e8969413cb7506200ced0b974]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:46.028350 20401 tablet_service.cc:1467] Processing CreateTablet for tablet 7afa0eba34a3419b85cd94a0fbe2f7c7 (DEFAULT_TABLE table=test-table [id=3296325e8969413cb7506200ced0b974]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:46.028824 20532 tablet_service.cc:1467] Processing CreateTablet for tablet 7afa0eba34a3419b85cd94a0fbe2f7c7 (DEFAULT_TABLE table=test-table [id=3296325e8969413cb7506200ced0b974]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:42:46.029465 20401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7afa0eba34a3419b85cd94a0fbe2f7c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:46.029467 20270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7afa0eba34a3419b85cd94a0fbe2f7c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:46.029783 20532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7afa0eba34a3419b85cd94a0fbe2f7c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:46.035625 20615 tablet_bootstrap.cc:492] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P 77547a85f3d94d139b01967920d1ea72: Bootstrap starting.
I20260927 02:42:46.036939 20617 tablet_bootstrap.cc:492] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P b3de8d38e8474754b3dc3c3d32f53ad3: Bootstrap starting.
I20260927 02:42:46.037288 20616 tablet_bootstrap.cc:492] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P 67a8864df8fa45a2bd043ba18d6be889: Bootstrap starting.
I20260927 02:42:46.037984 20615 tablet_bootstrap.cc:654] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P 77547a85f3d94d139b01967920d1ea72: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:46.038738 20615 log.cc:824] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P 77547a85f3d94d139b01967920d1ea72: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:46.039155 20617 tablet_bootstrap.cc:654] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P b3de8d38e8474754b3dc3c3d32f53ad3: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:46.039155 20616 tablet_bootstrap.cc:654] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P 67a8864df8fa45a2bd043ba18d6be889: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:46.039959 20617 log.cc:824] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P b3de8d38e8474754b3dc3c3d32f53ad3: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:46.039958 20616 log.cc:824] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P 67a8864df8fa45a2bd043ba18d6be889: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:46.041160 20615 tablet_bootstrap.cc:492] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P 77547a85f3d94d139b01967920d1ea72: No bootstrap required, opened a new log
I20260927 02:42:46.041338 20615 ts_tablet_manager.cc:1397] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P 77547a85f3d94d139b01967920d1ea72: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 02:42:46.041920 20616 tablet_bootstrap.cc:492] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P 67a8864df8fa45a2bd043ba18d6be889: No bootstrap required, opened a new log
I20260927 02:42:46.042104 20616 ts_tablet_manager.cc:1397] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P 67a8864df8fa45a2bd043ba18d6be889: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 02:42:46.042212 20617 tablet_bootstrap.cc:492] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P b3de8d38e8474754b3dc3c3d32f53ad3: No bootstrap required, opened a new log
I20260927 02:42:46.042407 20617 ts_tablet_manager.cc:1397] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P b3de8d38e8474754b3dc3c3d32f53ad3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260927 02:42:46.044512 20615 raft_consensus.cc:348] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P 77547a85f3d94d139b01967920d1ea72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a8864df8fa45a2bd043ba18d6be889" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33411 } } peers { permanent_uuid: "77547a85f3d94d139b01967920d1ea72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39525 } } peers { permanent_uuid: "b3de8d38e8474754b3dc3c3d32f53ad3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41637 } }
I20260927 02:42:46.044728 20615 raft_consensus.cc:374] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P 77547a85f3d94d139b01967920d1ea72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:46.044786 20615 raft_consensus.cc:729] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P 77547a85f3d94d139b01967920d1ea72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77547a85f3d94d139b01967920d1ea72, State: Initialized, Role: FOLLOWER
I20260927 02:42:46.045197 20615 consensus_queue.cc:260] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P 77547a85f3d94d139b01967920d1ea72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a8864df8fa45a2bd043ba18d6be889" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33411 } } peers { permanent_uuid: "77547a85f3d94d139b01967920d1ea72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39525 } } peers { permanent_uuid: "b3de8d38e8474754b3dc3c3d32f53ad3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41637 } }
I20260927 02:42:46.045812 20598 heartbeater.cc:503] Master 127.0.98.62:43957 was elected leader, sending a full tablet report...
I20260927 02:42:46.046072 20615 ts_tablet_manager.cc:1428] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P 77547a85f3d94d139b01967920d1ea72: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 02:42:46.046198 20616 raft_consensus.cc:348] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P 67a8864df8fa45a2bd043ba18d6be889 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a8864df8fa45a2bd043ba18d6be889" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33411 } } peers { permanent_uuid: "77547a85f3d94d139b01967920d1ea72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39525 } } peers { permanent_uuid: "b3de8d38e8474754b3dc3c3d32f53ad3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41637 } }
I20260927 02:42:46.046473 20616 raft_consensus.cc:374] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P 67a8864df8fa45a2bd043ba18d6be889 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:46.046885 20616 raft_consensus.cc:729] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P 67a8864df8fa45a2bd043ba18d6be889 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67a8864df8fa45a2bd043ba18d6be889, State: Initialized, Role: FOLLOWER
I20260927 02:42:46.046758 20617 raft_consensus.cc:348] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P b3de8d38e8474754b3dc3c3d32f53ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a8864df8fa45a2bd043ba18d6be889" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33411 } } peers { permanent_uuid: "77547a85f3d94d139b01967920d1ea72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39525 } } peers { permanent_uuid: "b3de8d38e8474754b3dc3c3d32f53ad3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41637 } }
I20260927 02:42:46.046979 20617 raft_consensus.cc:374] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P b3de8d38e8474754b3dc3c3d32f53ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:46.047040 20617 raft_consensus.cc:729] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P b3de8d38e8474754b3dc3c3d32f53ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3de8d38e8474754b3dc3c3d32f53ad3, State: Initialized, Role: FOLLOWER
I20260927 02:42:46.047356 20616 consensus_queue.cc:260] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P 67a8864df8fa45a2bd043ba18d6be889 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a8864df8fa45a2bd043ba18d6be889" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33411 } } peers { permanent_uuid: "77547a85f3d94d139b01967920d1ea72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39525 } } peers { permanent_uuid: "b3de8d38e8474754b3dc3c3d32f53ad3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41637 } }
I20260927 02:42:46.047526 20617 consensus_queue.cc:260] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P b3de8d38e8474754b3dc3c3d32f53ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a8864df8fa45a2bd043ba18d6be889" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33411 } } peers { permanent_uuid: "77547a85f3d94d139b01967920d1ea72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39525 } } peers { permanent_uuid: "b3de8d38e8474754b3dc3c3d32f53ad3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41637 } }
I20260927 02:42:46.048100 20467 heartbeater.cc:503] Master 127.0.98.62:43957 was elected leader, sending a full tablet report...
I20260927 02:42:46.048316 20616 ts_tablet_manager.cc:1428] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P 67a8864df8fa45a2bd043ba18d6be889: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 02:42:46.048655 20336 heartbeater.cc:503] Master 127.0.98.62:43957 was elected leader, sending a full tablet report...
I20260927 02:42:46.049293 20617 ts_tablet_manager.cc:1428] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P b3de8d38e8474754b3dc3c3d32f53ad3: Time spent starting tablet: real 0.007s user 0.003s sys 0.002s
W20260927 02:42:46.055790 20468 tablet.cc:2369] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P b3de8d38e8474754b3dc3c3d32f53ad3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:42:46.121004 20623 raft_consensus.cc:482] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P b3de8d38e8474754b3dc3c3d32f53ad3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:42:46.121258 20623 raft_consensus.cc:504] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P b3de8d38e8474754b3dc3c3d32f53ad3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a8864df8fa45a2bd043ba18d6be889" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33411 } } peers { permanent_uuid: "77547a85f3d94d139b01967920d1ea72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39525 } } peers { permanent_uuid: "b3de8d38e8474754b3dc3c3d32f53ad3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41637 } }
I20260927 02:42:46.122407 20623 leader_election.cc:288] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P b3de8d38e8474754b3dc3c3d32f53ad3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67a8864df8fa45a2bd043ba18d6be889 (127.0.98.1:33411), 77547a85f3d94d139b01967920d1ea72 (127.0.98.3:39525)
W20260927 02:42:46.123831 20337 tablet.cc:2369] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P 67a8864df8fa45a2bd043ba18d6be889: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:42:46.126226 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:46.122806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40333 (local address 127.0.98.1:33411)
0927 02:42:46.122951 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:46.122955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:46.122969 (+ 14us) server_negotiation.cc:407] Connection header received
0927 02:42:46.123023 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:46.123025 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:46.123072 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:46.123139 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:46.123459 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:46.124265 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:46.125330 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:46.125726 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:46.125820 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:46.125878 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:46.126044 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 02:42:46.126115 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 02:42:46.126227 20469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:46.122634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33411 (local address 127.0.98.2:40333)
0927 02:42:46.122773 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:46.122787 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:46.122882 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:46.123203 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:46.123206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:46.123226 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:46.123349 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:46.123356 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:46.124385 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:46.124388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:46.125208 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:46.125217 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:46.125837 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:46.125841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:46.125934 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:46.125951 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 02:42:46.126015 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 02:42:46.126083 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":3}
I20260927 02:42:46.126461 20624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:46.122895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39525 (local address 127.0.98.2:43623)
0927 02:42:46.123251 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:46.123264 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:42:46.123357 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:42:46.123582 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:42:46.123585 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:42:46.123601 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:42:46.123728 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:46.123735 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:46.124606 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:46.124610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:46.125400 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:42:46.125408 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:42:46.126040 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:42:46.126044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:42:46.126140 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:46.126158 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 02:42:46.126220 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 02:42:46.126305 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":18,"thread_start_us":124,"threads_started":1}
I20260927 02:42:46.126446 20614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:46.123070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43623 (local address 127.0.98.3:39525)
0927 02:42:46.123205 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:46.123208 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:46.123337 (+ 129us) server_negotiation.cc:407] Connection header received
0927 02:42:46.123443 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:46.123446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:46.123495 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:46.123589 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:42:46.123837 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:46.124498 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:46.125515 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:46.125903 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:46.125992 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:46.126047 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:46.126267 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 02:42:46.126338 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 02:42:46.126997 20552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afa0eba34a3419b85cd94a0fbe2f7c7" candidate_uuid: "b3de8d38e8474754b3dc3c3d32f53ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77547a85f3d94d139b01967920d1ea72" is_pre_election: true
I20260927 02:42:46.126998 20290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afa0eba34a3419b85cd94a0fbe2f7c7" candidate_uuid: "b3de8d38e8474754b3dc3c3d32f53ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67a8864df8fa45a2bd043ba18d6be889" is_pre_election: true
I20260927 02:42:46.127280 20290 raft_consensus.cc:2454] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P 67a8864df8fa45a2bd043ba18d6be889 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3de8d38e8474754b3dc3c3d32f53ad3 in term 0.
I20260927 02:42:46.127286 20552 raft_consensus.cc:2454] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P 77547a85f3d94d139b01967920d1ea72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3de8d38e8474754b3dc3c3d32f53ad3 in term 0.
I20260927 02:42:46.127820 20357 leader_election.cc:302] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P b3de8d38e8474754b3dc3c3d32f53ad3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77547a85f3d94d139b01967920d1ea72, b3de8d38e8474754b3dc3c3d32f53ad3; no voters:
I20260927 02:42:46.128093 20623 raft_consensus.cc:2781] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P b3de8d38e8474754b3dc3c3d32f53ad3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:42:46.128212 20623 raft_consensus.cc:482] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P b3de8d38e8474754b3dc3c3d32f53ad3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:42:46.128268 20623 raft_consensus.cc:3037] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P b3de8d38e8474754b3dc3c3d32f53ad3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:46.129336 20623 raft_consensus.cc:504] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P b3de8d38e8474754b3dc3c3d32f53ad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a8864df8fa45a2bd043ba18d6be889" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33411 } } peers { permanent_uuid: "77547a85f3d94d139b01967920d1ea72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39525 } } peers { permanent_uuid: "b3de8d38e8474754b3dc3c3d32f53ad3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41637 } }
I20260927 02:42:46.129644 20623 leader_election.cc:288] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P b3de8d38e8474754b3dc3c3d32f53ad3 [CANDIDATE]: Term 1 election: Requested vote from peers 67a8864df8fa45a2bd043ba18d6be889 (127.0.98.1:33411), 77547a85f3d94d139b01967920d1ea72 (127.0.98.3:39525)
I20260927 02:42:46.130066 20290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afa0eba34a3419b85cd94a0fbe2f7c7" candidate_uuid: "b3de8d38e8474754b3dc3c3d32f53ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67a8864df8fa45a2bd043ba18d6be889"
I20260927 02:42:46.130066 20552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afa0eba34a3419b85cd94a0fbe2f7c7" candidate_uuid: "b3de8d38e8474754b3dc3c3d32f53ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77547a85f3d94d139b01967920d1ea72"
I20260927 02:42:46.130198 20552 raft_consensus.cc:3037] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P 77547a85f3d94d139b01967920d1ea72 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:46.130198 20290 raft_consensus.cc:3037] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P 67a8864df8fa45a2bd043ba18d6be889 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:46.131464 20552 raft_consensus.cc:2454] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P 77547a85f3d94d139b01967920d1ea72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3de8d38e8474754b3dc3c3d32f53ad3 in term 1.
I20260927 02:42:46.131464 20290 raft_consensus.cc:2454] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P 67a8864df8fa45a2bd043ba18d6be889 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3de8d38e8474754b3dc3c3d32f53ad3 in term 1.
I20260927 02:42:46.131780 20357 leader_election.cc:302] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P b3de8d38e8474754b3dc3c3d32f53ad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77547a85f3d94d139b01967920d1ea72, b3de8d38e8474754b3dc3c3d32f53ad3; no voters:
I20260927 02:42:46.131945 20623 raft_consensus.cc:2781] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P b3de8d38e8474754b3dc3c3d32f53ad3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:46.132167 20623 raft_consensus.cc:686] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P b3de8d38e8474754b3dc3c3d32f53ad3 [term 1 LEADER]: Becoming Leader. State: Replica: b3de8d38e8474754b3dc3c3d32f53ad3, State: Running, Role: LEADER
I20260927 02:42:46.132462 20623 consensus_queue.cc:237] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P b3de8d38e8474754b3dc3c3d32f53ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a8864df8fa45a2bd043ba18d6be889" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33411 } } peers { permanent_uuid: "77547a85f3d94d139b01967920d1ea72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39525 } } peers { permanent_uuid: "b3de8d38e8474754b3dc3c3d32f53ad3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41637 } }
I20260927 02:42:46.135733 20148 catalog_manager.cc:5495] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P b3de8d38e8474754b3dc3c3d32f53ad3 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3de8d38e8474754b3dc3c3d32f53ad3 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b3de8d38e8474754b3dc3c3d32f53ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a8864df8fa45a2bd043ba18d6be889" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77547a85f3d94d139b01967920d1ea72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3de8d38e8474754b3dc3c3d32f53ad3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41637 } health_report { overall_health: HEALTHY } } }
I20260927 02:42:46.148133 20613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:46.144626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46116 (local address 127.0.98.2:41637)
0927 02:42:46.144768 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:46.144772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:46.144786 (+ 14us) server_negotiation.cc:407] Connection header received
0927 02:42:46.144916 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:46.144919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:46.144978 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:46.145072 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:42:46.145305 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:46.145812 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:46.146744 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:46.146966 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:46.147051 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:46.147342 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:42:46.147744 (+ 402us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:42:46.147853 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:46.147916 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 02:42:46.147968 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 02:42:46.153855 20290 raft_consensus.cc:1261] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P 67a8864df8fa45a2bd043ba18d6be889 [term 1 FOLLOWER]: Refusing update from remote peer b3de8d38e8474754b3dc3c3d32f53ad3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:42:46.153855 20552 raft_consensus.cc:1261] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P 77547a85f3d94d139b01967920d1ea72 [term 1 FOLLOWER]: Refusing update from remote peer b3de8d38e8474754b3dc3c3d32f53ad3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:42:46.154583 20623 consensus_queue.cc:1035] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P b3de8d38e8474754b3dc3c3d32f53ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77547a85f3d94d139b01967920d1ea72" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:42:46.154786 20625 consensus_queue.cc:1035] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P b3de8d38e8474754b3dc3c3d32f53ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a8864df8fa45a2bd043ba18d6be889" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:42:46.164726 20629 mvcc.cc:205] Tried to move back new op lower bound from 7333793653359140864 to 7333793653280968704. Current Snapshot: MvccSnapshot[applied={T|T < 7333793653359140864}]
I20260927 02:42:46.170066 20148 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:42:46.170208 20148 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:42:46.172489 20148 catalog_manager.cc:5782] T 00000000000000000000000000000000 P da91707e150e4e17973d356007f8d8c3: Sending DeleteTablet for 3 replicas of tablet 7afa0eba34a3419b85cd94a0fbe2f7c7
I20260927 02:42:46.173266 20401 tablet_service.cc:1514] Processing DeleteTablet for tablet 7afa0eba34a3419b85cd94a0fbe2f7c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:46 UTC) from {username='slave'} at 127.0.0.1:46112
I20260927 02:42:46.173430 20270 tablet_service.cc:1514] Processing DeleteTablet for tablet 7afa0eba34a3419b85cd94a0fbe2f7c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:46 UTC) from {username='slave'} at 127.0.0.1:42520
I20260927 02:42:46.173653 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 20208
I20260927 02:42:46.174188 20532 tablet_service.cc:1514] Processing DeleteTablet for tablet 7afa0eba34a3419b85cd94a0fbe2f7c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:42:46 UTC) from {username='slave'} at 127.0.0.1:53966
I20260927 02:42:46.178433 20639 tablet_replica.cc:330] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P 77547a85f3d94d139b01967920d1ea72: stopping tablet replica
I20260927 02:42:46.178767 20639 raft_consensus.cc:2229] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P 77547a85f3d94d139b01967920d1ea72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:42:46.179064 20639 raft_consensus.cc:2258] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P 77547a85f3d94d139b01967920d1ea72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:42:46.180325 20639 ts_tablet_manager.cc:1905] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P 77547a85f3d94d139b01967920d1ea72: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 02:42:46.181437 20135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33411 (error 108)
I20260927 02:42:46.181677 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 20339
I20260927 02:42:46.182926 20639 ts_tablet_manager.cc:1918] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P 77547a85f3d94d139b01967920d1ea72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:42:46.182991 20639 log.cc:1196] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P 77547a85f3d94d139b01967920d1ea72: Deleting WAL directory at /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790476718023131-392-0/minicluster-data/ts-2/wal/wals/7afa0eba34a3419b85cd94a0fbe2f7c7
I20260927 02:42:46.183245 20639 ts_tablet_manager.cc:1939] T 7afa0eba34a3419b85cd94a0fbe2f7c7 P 77547a85f3d94d139b01967920d1ea72: Deleting consensus metadata
I20260927 02:42:46.183651 20612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:46.183280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33411 (local address 127.0.0.1:42532)
0927 02:42:46.183493 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:46.183566 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260927 02:42:46.183826 20135 catalog_manager.cc:4584] TS 67a8864df8fa45a2bd043ba18d6be889 (127.0.98.1:33411): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7afa0eba34a3419b85cd94a0fbe2f7c7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33411: connect: Connection refused (error 111)
I20260927 02:42:46.184496 20135 catalog_manager.cc:4841] TS 77547a85f3d94d139b01967920d1ea72 (127.0.98.3:39525): tablet 7afa0eba34a3419b85cd94a0fbe2f7c7 (table test-table [id=3296325e8969413cb7506200ced0b974]) successfully deleted
I20260927 02:42:46.187722 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 20470
I20260927 02:42:46.187997 20612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:46.187736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41637 (local address 127.0.0.1:46122)
0927 02:42:46.187873 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 02:42:46.187925 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 02:42:46.192178 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 20117
2026-09-27T02:42:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (992 ms)
[----------] 6 tests from EncryptionPolicyTest (9821 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476718023131-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:42:46 dist-test-slave-zv8z krb5kdc[20643](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:42:46 dist-test-slave-zv8z krb5kdc[20643](info): set up 2 sockets
Sep 27 02:42:46 dist-test-slave-zv8z krb5kdc[20643](info): commencing operation
krb5kdc: starting...
W20260927 02:42:47.146034 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476718023131-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476718023131-392-0/krb5kdc/test-user.keytab.
Sep 27 02:42:47 dist-test-slave-zv8z krb5kdc[20643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:42:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:42:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476718023131-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476718023131-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:42:47.258937 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34475
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476718023131-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:47.364200 20656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:47.364483 20656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:47.364542 20656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:47.368587 20656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:42:47.368649 20656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:47.368676 20656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:47.368696 20656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:42:47.368781 20656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:42:47.374073 20656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34475
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476718023131-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:42:47.375129 20656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:47.376122 20656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:47.384146 20661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:47.384166 20662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:47.384193 20656 server_base.cc:1034] running on GCE node
W20260927 02:42:47.384223 20664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:47.384959 20656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:47.385984 20656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:47.387146 20656 hybrid_clock.cc:648] HybridClock initialized: now 1790476967387117 us; error 45 us; skew 500 ppm
Sep 27 02:42:47 dist-test-slave-zv8z krb5kdc[20643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:47.393092 20656 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:42:47.393940 20656 webserver.cc:460] Webserver started at http://127.0.0.1:46185/ using document root <none> and password file <none>
I20260927 02:42:47.394506 20656 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:47.394586 20656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:47.394802 20656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:47.396759 20656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476718023131-392-0/minicluster-data/master-0/data/instance:
uuid: "46971b07931b4d3ca537c411fe98d88b"
format_stamp: "Formatted at 2026-09-27 02:42:47 on dist-test-slave-zv8z"
server_key: "e35aaa6b3f2a21fb3d9cb6b21d8110db"
server_key_iv: "27f1d03f7e13b4ab5d6812c0b2e1a0c8"
server_key_version: "encryptionkey@0"
I20260927 02:42:47.397274 20656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476718023131-392-0/minicluster-data/master-0/wal/instance:
uuid: "46971b07931b4d3ca537c411fe98d88b"
format_stamp: "Formatted at 2026-09-27 02:42:47 on dist-test-slave-zv8z"
server_key: "e35aaa6b3f2a21fb3d9cb6b21d8110db"
server_key_iv: "27f1d03f7e13b4ab5d6812c0b2e1a0c8"
server_key_version: "encryptionkey@0"
I20260927 02:42:47.400846 20656 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 02:42:47.403440 20671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:47.404744 20656 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:42:47.404860 20656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "46971b07931b4d3ca537c411fe98d88b"
format_stamp: "Formatted at 2026-09-27 02:42:47 on dist-test-slave-zv8z"
server_key: "e35aaa6b3f2a21fb3d9cb6b21d8110db"
server_key_iv: "27f1d03f7e13b4ab5d6812c0b2e1a0c8"
server_key_version: "encryptionkey@0"
I20260927 02:42:47.404955 20656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:47.424168 20656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:47.426967 20656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:47.427140 20656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:47.433707 20656 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34475
I20260927 02:42:47.433732 20723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34475 every 8 connection(s)
I20260927 02:42:47.434638 20656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:42:47.437268 20724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:47.442629 20724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46971b07931b4d3ca537c411fe98d88b: Bootstrap starting.
I20260927 02:42:47.444341 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 20656
I20260927 02:42:47.444479 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790476718023131-392-0/minicluster-data/master-0/wal/instance
I20260927 02:42:47.444769 392 external_mini_cluster.cc:1428] Setting key c970804115000bd117b69c9837ab3af1
I20260927 02:42:47.445278 20724 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46971b07931b4d3ca537c411fe98d88b: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:47.446091 20724 log.cc:824] T 00000000000000000000000000000000 P 46971b07931b4d3ca537c411fe98d88b: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:47.448035 20724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46971b07931b4d3ca537c411fe98d88b: No bootstrap required, opened a new log
I20260927 02:42:47.451435 20724 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46971b07931b4d3ca537c411fe98d88b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46971b07931b4d3ca537c411fe98d88b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34475 } }
I20260927 02:42:47.451685 20724 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46971b07931b4d3ca537c411fe98d88b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:47.451747 20724 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46971b07931b4d3ca537c411fe98d88b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46971b07931b4d3ca537c411fe98d88b, State: Initialized, Role: FOLLOWER
I20260927 02:42:47.452148 20724 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46971b07931b4d3ca537c411fe98d88b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46971b07931b4d3ca537c411fe98d88b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34475 } }
I20260927 02:42:47.452270 20724 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46971b07931b4d3ca537c411fe98d88b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:42:47.452337 20724 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46971b07931b4d3ca537c411fe98d88b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:42:47.452427 20724 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46971b07931b4d3ca537c411fe98d88b [term 0 FOLLOWER]: Advancing to term 1
Sep 27 02:42:47 dist-test-slave-zv8z krb5kdc[20643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 02:42:47.453271 20724 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46971b07931b4d3ca537c411fe98d88b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46971b07931b4d3ca537c411fe98d88b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34475 } }
I20260927 02:42:47.453608 20724 leader_election.cc:302] T 00000000000000000000000000000000 P 46971b07931b4d3ca537c411fe98d88b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46971b07931b4d3ca537c411fe98d88b; no voters:
I20260927 02:42:47.453951 20724 leader_election.cc:288] T 00000000000000000000000000000000 P 46971b07931b4d3ca537c411fe98d88b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:42:47.454095 20729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46971b07931b4d3ca537c411fe98d88b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:47.454327 20729 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46971b07931b4d3ca537c411fe98d88b [term 1 LEADER]: Becoming Leader. State: Replica: 46971b07931b4d3ca537c411fe98d88b, State: Running, Role: LEADER
I20260927 02:42:47.454743 20729 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46971b07931b4d3ca537c411fe98d88b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46971b07931b4d3ca537c411fe98d88b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34475 } }
I20260927 02:42:47.455288 20724 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46971b07931b4d3ca537c411fe98d88b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:42:47.455763 20729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46971b07931b4d3ca537c411fe98d88b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46971b07931b4d3ca537c411fe98d88b. Latest consensus state: current_term: 1 leader_uuid: "46971b07931b4d3ca537c411fe98d88b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46971b07931b4d3ca537c411fe98d88b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34475 } } }
I20260927 02:42:47.455873 20729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46971b07931b4d3ca537c411fe98d88b [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:47.456292 20730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46971b07931b4d3ca537c411fe98d88b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46971b07931b4d3ca537c411fe98d88b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46971b07931b4d3ca537c411fe98d88b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34475 } } }
I20260927 02:42:47.456475 20730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46971b07931b4d3ca537c411fe98d88b [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:47.456346 20734 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:42:47.459226 20727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:47.445862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51177 (local address 127.0.0.1:34475)
0927 02:42:47.446335 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:47.446351 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:47.446385 (+ 34us) server_negotiation.cc:407] Connection header received
0927 02:42:47.447112 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:47.447131 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:47.447472 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:47.447773 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:47.448532 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:47.449487 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:47.450334 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:47.450791 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:47.450928 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:47.453529 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:47.453535 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:47.453551 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:47.453581 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:47.455639 (+ 2058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:47.456288 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:47.456294 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:47.456300 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:47.456441 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:47.456883 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:47.456886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:47.456888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:47.457417 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:47.457712 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:47.458049 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 02:42:47.458296 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":134,"threads_started":1}
I20260927 02:42:47.459662 20734 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:42:47.462308 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:42:47.465749 20734 catalog_manager.cc:1347] Generated new cluster ID: ccbf3b23c6a04c30afc4cb3be472aab6
I20260927 02:42:47.465809 20734 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:42:47.473506 20727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:47.464118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45129 (local address 127.0.0.1:34475)
0927 02:42:47.464217 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:47.464221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:47.464742 (+ 521us) server_negotiation.cc:407] Connection header received
0927 02:42:47.464849 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:47.464851 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:47.464896 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:47.464976 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:47.465760 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:47.466589 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:47.467761 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:47.468149 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:47.468221 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:47.469989 (+ 1768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:47.469993 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:47.469996 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:47.470020 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:47.471515 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:47.472090 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:47.472094 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:47.472096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:47.472177 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:47.472538 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:47.472541 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:47.472543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:47.472752 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:47.472865 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:47.473252 (+ 387us) server_negotiation.cc:299] Negotiation successful
0927 02:42:47.473343 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 02:42:47.499675 20734 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:42:47.500612 20734 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:42:47.505116 20734 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46971b07931b4d3ca537c411fe98d88b: Generated new TSK 0
I20260927 02:42:47.505769 20734 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:42:47.515515 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 20656
W20260927 02:42:47.520118 20641 connection.cc:535] client connection to 127.0.0.1:34475 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34475): Connection reset by peer (error 104)
2026-09-27T02:42:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1321 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476718023131-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:42:47 dist-test-slave-zv8z krb5kdc[20755](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:42:47 dist-test-slave-zv8z krb5kdc[20755](info): set up 2 sockets
Sep 27 02:42:47 dist-test-slave-zv8z krb5kdc[20755](info): commencing operation
krb5kdc: starting...
W20260927 02:42:48.462133 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476718023131-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476718023131-392-0/krb5kdc/test-user.keytab.
Sep 27 02:42:48 dist-test-slave-zv8z krb5kdc[20755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:42:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:42:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476718023131-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476718023131-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:42:48.577000 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34247
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476718023131-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:48.682538 20768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:48.682904 20768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:48.682976 20768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:48.687361 20768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:42:48.687420 20768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:48.687438 20768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:48.687453 20768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:42:48.687467 20768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:42:48.692762 20768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34247
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476718023131-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:42:48.693799 20768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:48.694832 20768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:48.702945 20776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:48.702986 20773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:48.702975 20774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:48.703100 20768 server_base.cc:1034] running on GCE node
I20260927 02:42:48.703665 20768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:48.704701 20768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:48.705859 20768 hybrid_clock.cc:648] HybridClock initialized: now 1790476968705843 us; error 30 us; skew 500 ppm
Sep 27 02:42:48 dist-test-slave-zv8z krb5kdc[20755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:48.708515 20768 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:42:48.709376 20768 webserver.cc:460] Webserver started at http://127.0.0.1:40699/ using document root <none> and password file <none>
I20260927 02:42:48.709897 20768 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:48.709960 20768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:48.710139 20768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:48.712150 20768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476718023131-392-0/minicluster-data/master-0/data/instance:
uuid: "063739147930468e98cd110b53841cf9"
format_stamp: "Formatted at 2026-09-27 02:42:48 on dist-test-slave-zv8z"
server_key: "9d4009aa7607a47061c4f004daa49e5f"
server_key_iv: "896accae60c5052897f9863778ff151e"
server_key_version: "encryptionkey@0"
I20260927 02:42:48.712661 20768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476718023131-392-0/minicluster-data/master-0/wal/instance:
uuid: "063739147930468e98cd110b53841cf9"
format_stamp: "Formatted at 2026-09-27 02:42:48 on dist-test-slave-zv8z"
server_key: "9d4009aa7607a47061c4f004daa49e5f"
server_key_iv: "896accae60c5052897f9863778ff151e"
server_key_version: "encryptionkey@0"
I20260927 02:42:48.716248 20768 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:42:48.718541 20783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:48.719709 20768 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 02:42:48.719822 20768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "063739147930468e98cd110b53841cf9"
format_stamp: "Formatted at 2026-09-27 02:42:48 on dist-test-slave-zv8z"
server_key: "9d4009aa7607a47061c4f004daa49e5f"
server_key_iv: "896accae60c5052897f9863778ff151e"
server_key_version: "encryptionkey@0"
I20260927 02:42:48.719913 20768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:48.738597 20768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:48.741160 20768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:48.741330 20768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:48.747776 20768 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34247
I20260927 02:42:48.747833 20835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34247 every 8 connection(s)
I20260927 02:42:48.748673 20768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:42:48.751576 20836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:48.752372 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 20768
I20260927 02:42:48.752480 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790476718023131-392-0/minicluster-data/master-0/wal/instance
I20260927 02:42:48.752748 392 external_mini_cluster.cc:1428] Setting key b76a23805c2d8e5a4beeda2ef08eb475
I20260927 02:42:48.757711 20836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 063739147930468e98cd110b53841cf9: Bootstrap starting.
I20260927 02:42:48.760084 20836 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 063739147930468e98cd110b53841cf9: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:48.760871 20836 log.cc:824] T 00000000000000000000000000000000 P 063739147930468e98cd110b53841cf9: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:48.762310 20839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:48.754060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52139 (local address 127.0.0.1:34247)
0927 02:42:48.754471 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:48.754489 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:48.754546 (+ 57us) server_negotiation.cc:407] Connection header received
0927 02:42:48.755317 (+ 771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:48.755340 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:48.755759 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:48.756121 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:48.756500 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:48.757920 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:48.758829 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:48.759481 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:48.759633 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:48.760431 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:48.760437 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:48.760450 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:42:48.760928 (+ 478us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:48.760992 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:42:48.761021 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:48.761142 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:48.761287 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 02:42:48.761429 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":119,"threads_started":1}
I20260927 02:42:48.762943 20836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 063739147930468e98cd110b53841cf9: No bootstrap required, opened a new log
I20260927 02:42:48.765123 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:42:48.766484 20836 raft_consensus.cc:348] T 00000000000000000000000000000000 P 063739147930468e98cd110b53841cf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "063739147930468e98cd110b53841cf9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34247 } }
I20260927 02:42:48.766701 20836 raft_consensus.cc:374] T 00000000000000000000000000000000 P 063739147930468e98cd110b53841cf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:48.766764 20836 raft_consensus.cc:729] T 00000000000000000000000000000000 P 063739147930468e98cd110b53841cf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 063739147930468e98cd110b53841cf9, State: Initialized, Role: FOLLOWER
I20260927 02:42:48.767235 20836 consensus_queue.cc:260] T 00000000000000000000000000000000 P 063739147930468e98cd110b53841cf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "063739147930468e98cd110b53841cf9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34247 } }
I20260927 02:42:48.767370 20836 raft_consensus.cc:388] T 00000000000000000000000000000000 P 063739147930468e98cd110b53841cf9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:42:48.767431 20836 raft_consensus.cc:482] T 00000000000000000000000000000000 P 063739147930468e98cd110b53841cf9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:42:48.767530 20836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 063739147930468e98cd110b53841cf9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:48.768335 20836 raft_consensus.cc:504] T 00000000000000000000000000000000 P 063739147930468e98cd110b53841cf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "063739147930468e98cd110b53841cf9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34247 } }
I20260927 02:42:48.768630 20836 leader_election.cc:302] T 00000000000000000000000000000000 P 063739147930468e98cd110b53841cf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 063739147930468e98cd110b53841cf9; no voters:
I20260927 02:42:48.768896 20836 leader_election.cc:288] T 00000000000000000000000000000000 P 063739147930468e98cd110b53841cf9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:42:48.768975 20846 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 063739147930468e98cd110b53841cf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:48.769129 20846 raft_consensus.cc:686] T 00000000000000000000000000000000 P 063739147930468e98cd110b53841cf9 [term 1 LEADER]: Becoming Leader. State: Replica: 063739147930468e98cd110b53841cf9, State: Running, Role: LEADER
I20260927 02:42:48.769403 20846 consensus_queue.cc:237] T 00000000000000000000000000000000 P 063739147930468e98cd110b53841cf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "063739147930468e98cd110b53841cf9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34247 } }
I20260927 02:42:48.770082 20836 sys_catalog.cc:564] T 00000000000000000000000000000000 P 063739147930468e98cd110b53841cf9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:42:48.770560 20846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 063739147930468e98cd110b53841cf9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 063739147930468e98cd110b53841cf9. Latest consensus state: current_term: 1 leader_uuid: "063739147930468e98cd110b53841cf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "063739147930468e98cd110b53841cf9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34247 } } }
I20260927 02:42:48.770668 20846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 063739147930468e98cd110b53841cf9 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:48.770846 20839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:48.766916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48065 (local address 127.0.0.1:34247)
0927 02:42:48.767010 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:48.767014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:48.767184 (+ 170us) server_negotiation.cc:407] Connection header received
0927 02:42:48.767289 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:48.767292 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:48.767336 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:48.767400 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:48.767679 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:48.768166 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:48.769043 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:48.769369 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:48.769453 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:48.770360 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:48.770364 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:48.770367 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:42:48.770379 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:48.770403 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:42:48.770415 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:48.770486 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:48.770592 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 02:42:48.770621 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 02:42:48.771250 20847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 063739147930468e98cd110b53841cf9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "063739147930468e98cd110b53841cf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "063739147930468e98cd110b53841cf9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34247 } } }
I20260927 02:42:48.771340 20847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 063739147930468e98cd110b53841cf9 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:48.771741 20851 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:42:48.774128 20851 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:42:48.779667 20851 catalog_manager.cc:1347] Generated new cluster ID: 9f3a7ac5fc5a43e5ae109f1bde5f6a68
I20260927 02:42:48.779726 20851 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:42:48.793943 20851 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:42:48.794961 20851 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:42:48.800668 20851 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 063739147930468e98cd110b53841cf9: Generated new TSK 0
I20260927 02:42:48.801440 20851 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:42:48.815549 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 20768
W20260927 02:42:48.820176 20753 connection.cc:535] client connection to 127.0.0.1:34247 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34247): Connection reset by peer (error 104)
2026-09-27T02:42:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1300 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476718023131-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:42:48 dist-test-slave-zv8z krb5kdc[20867](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:42:48 dist-test-slave-zv8z krb5kdc[20867](info): set up 2 sockets
Sep 27 02:42:48 dist-test-slave-zv8z krb5kdc[20867](info): commencing operation
krb5kdc: starting...
W20260927 02:42:49.762157 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476718023131-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476718023131-392-0/krb5kdc/test-user.keytab.
Sep 27 02:42:49 dist-test-slave-zv8z krb5kdc[20867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:42:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:42:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476718023131-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476718023131-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:42:49.875406 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46471
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476718023131-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:49.980490 20880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:49.980752 20880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:49.980791 20880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:49.984692 20880 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:42:49.984748 20880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:49.984766 20880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:49.984781 20880 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:42:49.984794 20880 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:42:49.989892 20880 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46471
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476718023131-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:42:49.990918 20880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:49.991858 20880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:50.000168 20886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:50.000151 20885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:50.000171 20888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:50.000581 20880 server_base.cc:1034] running on GCE node
I20260927 02:42:50.001181 20880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:50.002327 20880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:50.003495 20880 hybrid_clock.cc:648] HybridClock initialized: now 1790476970003469 us; error 35 us; skew 500 ppm
Sep 27 02:42:49 dist-test-slave-zv8z krb5kdc[20867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476969, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:50.006385 20880 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:42:50.007236 20880 webserver.cc:460] Webserver started at http://127.0.0.1:36339/ using document root <none> and password file <none>
I20260927 02:42:50.007766 20880 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:50.007830 20880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:50.008008 20880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:50.009989 20880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476718023131-392-0/minicluster-data/master-0/data/instance:
uuid: "25362157e48a41cc9930c32e446421c1"
format_stamp: "Formatted at 2026-09-27 02:42:50 on dist-test-slave-zv8z"
server_key: "f3500f2e929bdfcd399bdb903140bb35"
server_key_iv: "ea2d2570b202a302c5fd44ad273e7786"
server_key_version: "encryptionkey@0"
I20260927 02:42:50.010536 20880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476718023131-392-0/minicluster-data/master-0/wal/instance:
uuid: "25362157e48a41cc9930c32e446421c1"
format_stamp: "Formatted at 2026-09-27 02:42:50 on dist-test-slave-zv8z"
server_key: "f3500f2e929bdfcd399bdb903140bb35"
server_key_iv: "ea2d2570b202a302c5fd44ad273e7786"
server_key_version: "encryptionkey@0"
I20260927 02:42:50.014031 20880 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 02:42:50.016355 20895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:50.017606 20880 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:42:50.017726 20880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "25362157e48a41cc9930c32e446421c1"
format_stamp: "Formatted at 2026-09-27 02:42:50 on dist-test-slave-zv8z"
server_key: "f3500f2e929bdfcd399bdb903140bb35"
server_key_iv: "ea2d2570b202a302c5fd44ad273e7786"
server_key_version: "encryptionkey@0"
I20260927 02:42:50.017823 20880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:50.022305 20880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:50.025385 20880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:50.025533 20880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:50.032096 20880 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46471
I20260927 02:42:50.032143 20947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46471 every 8 connection(s)
I20260927 02:42:50.033018 20880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:42:50.035768 20948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:50.041052 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 20880
I20260927 02:42:50.041097 20948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25362157e48a41cc9930c32e446421c1: Bootstrap starting.
I20260927 02:42:50.041188 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790476718023131-392-0/minicluster-data/master-0/wal/instance
I20260927 02:42:50.041414 392 external_mini_cluster.cc:1428] Setting key d97a2504b8b1f5e713b1f1ba1b6a911f
I20260927 02:42:50.043625 20948 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25362157e48a41cc9930c32e446421c1: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:50.044523 20948 log.cc:824] T 00000000000000000000000000000000 P 25362157e48a41cc9930c32e446421c1: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:50.046651 20948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25362157e48a41cc9930c32e446421c1: No bootstrap required, opened a new log
I20260927 02:42:50.046949 20951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:50.042759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38553 (local address 127.0.0.1:46471)
0927 02:42:50.043187 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:50.043205 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:50.043247 (+ 42us) server_negotiation.cc:407] Connection header received
0927 02:42:50.044077 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:50.044100 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:50.044495 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:50.044884 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:50.045649 (+ 765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:50.045654 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:50.045672 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:42:50.045682 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:50.045732 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:42:50.045752 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:50.045831 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:50.045945 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 02:42:50.046087 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":120,"threads_started":1}
I20260927 02:42:50.049589 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:42:50.050460 20948 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25362157e48a41cc9930c32e446421c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25362157e48a41cc9930c32e446421c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46471 } }
I20260927 02:42:50.050722 20948 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25362157e48a41cc9930c32e446421c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:50.050781 20948 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25362157e48a41cc9930c32e446421c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25362157e48a41cc9930c32e446421c1, State: Initialized, Role: FOLLOWER
I20260927 02:42:50.051193 20948 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25362157e48a41cc9930c32e446421c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25362157e48a41cc9930c32e446421c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46471 } }
I20260927 02:42:50.051321 20948 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25362157e48a41cc9930c32e446421c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:42:50.051405 20948 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25362157e48a41cc9930c32e446421c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:42:50.051499 20948 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25362157e48a41cc9930c32e446421c1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:50.052402 20948 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25362157e48a41cc9930c32e446421c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25362157e48a41cc9930c32e446421c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46471 } }
I20260927 02:42:50.052732 20948 leader_election.cc:302] T 00000000000000000000000000000000 P 25362157e48a41cc9930c32e446421c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25362157e48a41cc9930c32e446421c1; no voters:
I20260927 02:42:50.052990 20951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:50.051438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41217 (local address 127.0.0.1:46471)
0927 02:42:50.051531 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:50.051534 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:50.051787 (+ 253us) server_negotiation.cc:407] Connection header received
0927 02:42:50.051889 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:50.051892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:50.051923 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:50.051982 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:50.052635 (+ 653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:50.052639 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:50.052641 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:42:50.052652 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:50.052685 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:42:50.052695 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:50.052761 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:50.052844 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 02:42:50.052881 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 02:42:50.053001 20948 leader_election.cc:288] T 00000000000000000000000000000000 P 25362157e48a41cc9930c32e446421c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:42:50.053063 20958 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25362157e48a41cc9930c32e446421c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:50.053751 20958 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25362157e48a41cc9930c32e446421c1 [term 1 LEADER]: Becoming Leader. State: Replica: 25362157e48a41cc9930c32e446421c1, State: Running, Role: LEADER
I20260927 02:42:50.054059 20958 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25362157e48a41cc9930c32e446421c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25362157e48a41cc9930c32e446421c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46471 } }
I20260927 02:42:50.054157 20948 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25362157e48a41cc9930c32e446421c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:42:50.055634 20960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25362157e48a41cc9930c32e446421c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25362157e48a41cc9930c32e446421c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25362157e48a41cc9930c32e446421c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46471 } } }
I20260927 02:42:50.055751 20960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25362157e48a41cc9930c32e446421c1 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:50.056006 20959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25362157e48a41cc9930c32e446421c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25362157e48a41cc9930c32e446421c1. Latest consensus state: current_term: 1 leader_uuid: "25362157e48a41cc9930c32e446421c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25362157e48a41cc9930c32e446421c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46471 } } }
I20260927 02:42:50.056442 20959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25362157e48a41cc9930c32e446421c1 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:50.056473 20972 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:42:50.059091 20972 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:42:50.064105 20972 catalog_manager.cc:1347] Generated new cluster ID: a1f1bc5bfade4b6f9531151105d2d6a1
I20260927 02:42:50.064163 20972 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:42:50.081321 20972 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:42:50.081626 20972 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:42:50.097939 20972 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25362157e48a41cc9930c32e446421c1: Generated new TSK 0
I20260927 02:42:50.098671 20972 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:42:50.109464 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 20880
2026-09-27T02:42:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1293 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476718023131-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:42:50 dist-test-slave-zv8z krb5kdc[20979](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:42:50 dist-test-slave-zv8z krb5kdc[20979](info): set up 2 sockets
Sep 27 02:42:50 dist-test-slave-zv8z krb5kdc[20979](info): commencing operation
krb5kdc: starting...
W20260927 02:42:51.066154 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476718023131-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476718023131-392-0/krb5kdc/test-user.keytab.
Sep 27 02:42:51 dist-test-slave-zv8z krb5kdc[20979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:42:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:42:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476718023131-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476718023131-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:42:51.177258 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33147
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476718023131-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33147
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:51.282400 20992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:51.282657 20992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:51.282714 20992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:51.286698 20992 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:42:51.286756 20992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:51.286782 20992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:51.286804 20992 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:42:51.286834 20992 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:42:51.292048 20992 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476718023131-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:42:51.293040 20992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:51.293999 20992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:51.302121 20997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:51.302268 20992 server_base.cc:1034] running on GCE node
W20260927 02:42:51.302150 21000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:51.302433 20998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:51.302970 20992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:51.304040 20992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:51.305195 20992 hybrid_clock.cc:648] HybridClock initialized: now 1790476971305177 us; error 30 us; skew 500 ppm
Sep 27 02:42:51 dist-test-slave-zv8z krb5kdc[20979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:51.307886 20992 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:42:51.308794 20992 webserver.cc:460] Webserver started at http://127.0.0.1:44291/ using document root <none> and password file <none>
I20260927 02:42:51.309303 20992 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:51.309365 20992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:51.309559 20992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:51.311537 20992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476718023131-392-0/minicluster-data/master-0/data/instance:
uuid: "b2a8d5c1f69e4937b0bbfca280726b94"
format_stamp: "Formatted at 2026-09-27 02:42:51 on dist-test-slave-zv8z"
server_key: "572e4d7ef11a47b4a498b71fa1751e7e"
server_key_iv: "96eaa80a881405e9924ff5c53b825110"
server_key_version: "encryptionkey@0"
I20260927 02:42:51.312047 20992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476718023131-392-0/minicluster-data/master-0/wal/instance:
uuid: "b2a8d5c1f69e4937b0bbfca280726b94"
format_stamp: "Formatted at 2026-09-27 02:42:51 on dist-test-slave-zv8z"
server_key: "572e4d7ef11a47b4a498b71fa1751e7e"
server_key_iv: "96eaa80a881405e9924ff5c53b825110"
server_key_version: "encryptionkey@0"
I20260927 02:42:51.315443 20992 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:42:51.317704 21007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:51.318909 20992 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:42:51.319023 20992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "b2a8d5c1f69e4937b0bbfca280726b94"
format_stamp: "Formatted at 2026-09-27 02:42:51 on dist-test-slave-zv8z"
server_key: "572e4d7ef11a47b4a498b71fa1751e7e"
server_key_iv: "96eaa80a881405e9924ff5c53b825110"
server_key_version: "encryptionkey@0"
I20260927 02:42:51.319108 20992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:51.344945 20992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:51.348147 20992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:51.348323 20992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:51.355288 20992 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33147
I20260927 02:42:51.355304 21059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33147 every 8 connection(s)
I20260927 02:42:51.356189 20992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:42:51.358877 21060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:51.362840 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 20992
I20260927 02:42:51.362978 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790476718023131-392-0/minicluster-data/master-0/wal/instance
I20260927 02:42:51.363293 392 external_mini_cluster.cc:1428] Setting key 7d046754db306d9e8eb29d358b5f3454
I20260927 02:42:51.364684 21060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2a8d5c1f69e4937b0bbfca280726b94: Bootstrap starting.
I20260927 02:42:51.367066 21060 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2a8d5c1f69e4937b0bbfca280726b94: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:51.367805 21060 log.cc:824] T 00000000000000000000000000000000 P b2a8d5c1f69e4937b0bbfca280726b94: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:51.369697 21060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2a8d5c1f69e4937b0bbfca280726b94: No bootstrap required, opened a new log
Sep 27 02:42:51 dist-test-slave-zv8z krb5kdc[20979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 02:42:51.373726 21060 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2a8d5c1f69e4937b0bbfca280726b94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2a8d5c1f69e4937b0bbfca280726b94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33147 } }
I20260927 02:42:51.373978 21060 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2a8d5c1f69e4937b0bbfca280726b94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:51.374042 21060 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2a8d5c1f69e4937b0bbfca280726b94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2a8d5c1f69e4937b0bbfca280726b94, State: Initialized, Role: FOLLOWER
I20260927 02:42:51.374528 21060 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2a8d5c1f69e4937b0bbfca280726b94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2a8d5c1f69e4937b0bbfca280726b94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33147 } }
I20260927 02:42:51.374663 21060 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2a8d5c1f69e4937b0bbfca280726b94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:42:51.374734 21060 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2a8d5c1f69e4937b0bbfca280726b94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:42:51.374828 21060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2a8d5c1f69e4937b0bbfca280726b94 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:51.375608 21060 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2a8d5c1f69e4937b0bbfca280726b94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2a8d5c1f69e4937b0bbfca280726b94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33147 } }
I20260927 02:42:51.375926 21060 leader_election.cc:302] T 00000000000000000000000000000000 P b2a8d5c1f69e4937b0bbfca280726b94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2a8d5c1f69e4937b0bbfca280726b94; no voters:
I20260927 02:42:51.376231 21060 leader_election.cc:288] T 00000000000000000000000000000000 P b2a8d5c1f69e4937b0bbfca280726b94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:42:51.376312 21065 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2a8d5c1f69e4937b0bbfca280726b94 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:51.376497 21065 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2a8d5c1f69e4937b0bbfca280726b94 [term 1 LEADER]: Becoming Leader. State: Replica: b2a8d5c1f69e4937b0bbfca280726b94, State: Running, Role: LEADER
I20260927 02:42:51.376799 21065 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2a8d5c1f69e4937b0bbfca280726b94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2a8d5c1f69e4937b0bbfca280726b94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33147 } }
I20260927 02:42:51.377305 21063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:51.364619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46711 (local address 127.0.0.1:33147)
0927 02:42:51.365010 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:51.365025 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:51.365060 (+ 35us) server_negotiation.cc:407] Connection header received
0927 02:42:51.365697 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:51.365713 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:51.366054 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:51.366368 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:51.367247 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:51.368245 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:51.369110 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:51.369789 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:51.369940 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:51.372432 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:51.372439 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:51.372455 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:51.372488 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:51.374301 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:51.374709 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:51.374717 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:51.374723 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:51.374880 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:51.375136 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:51.375139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:51.375141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:51.375528 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:51.375714 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:51.375978 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 02:42:51.376288 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":111,"threads_started":1}
I20260927 02:42:51.377389 21060 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2a8d5c1f69e4937b0bbfca280726b94 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:42:51.377965 21065 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2a8d5c1f69e4937b0bbfca280726b94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2a8d5c1f69e4937b0bbfca280726b94. Latest consensus state: current_term: 1 leader_uuid: "b2a8d5c1f69e4937b0bbfca280726b94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2a8d5c1f69e4937b0bbfca280726b94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33147 } } }
I20260927 02:42:51.378070 21065 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2a8d5c1f69e4937b0bbfca280726b94 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:51.378376 21066 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2a8d5c1f69e4937b0bbfca280726b94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2a8d5c1f69e4937b0bbfca280726b94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2a8d5c1f69e4937b0bbfca280726b94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33147 } } }
I20260927 02:42:51.378463 21066 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2a8d5c1f69e4937b0bbfca280726b94 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:51.378661 21071 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:42:51.380506 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:42:51.381770 21071 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:42:51.387318 21071 catalog_manager.cc:1347] Generated new cluster ID: 8b9d0e418cc14a4aa1e1280740e164e9
I20260927 02:42:51.387382 21071 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:42:51.390687 21063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:51.382648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38925 (local address 127.0.0.1:33147)
0927 02:42:51.382757 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:51.382761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:51.383159 (+ 398us) server_negotiation.cc:407] Connection header received
0927 02:42:51.383258 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:51.383261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:51.383303 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:51.383362 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:51.384100 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:51.384921 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:51.385963 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:51.386370 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:51.386477 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:51.387960 (+ 1483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:51.387963 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:51.387966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:51.387990 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:51.389247 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:51.389605 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:51.389608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:51.389610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:51.389681 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:51.389918 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:51.389921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:51.389923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:51.390083 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:51.390190 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:51.390446 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 02:42:51.390547 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 02:42:51.394919 21071 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:42:51.395826 21071 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:42:51.403020 21071 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2a8d5c1f69e4937b0bbfca280726b94: Generated new TSK 0
I20260927 02:42:51.403753 21071 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:42:51.407555 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 20992
2026-09-27T02:42:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1297 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476718023131-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:42:51 dist-test-slave-zv8z krb5kdc[21091](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:42:51 dist-test-slave-zv8z krb5kdc[21091](info): set up 2 sockets
Sep 27 02:42:51 dist-test-slave-zv8z krb5kdc[21091](info): commencing operation
krb5kdc: starting...
W20260927 02:42:52.350108 392 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476718023131-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476718023131-392-0/krb5kdc/test-user.keytab.
Sep 27 02:42:52 dist-test-slave-zv8z krb5kdc[21091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:42:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:42:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476718023131-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476718023131-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:42:52.461297 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46069
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476718023131-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:52.565968 21104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:52.566228 21104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:52.566306 21104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:52.570199 21104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:42:52.570276 21104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:52.570305 21104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:52.570330 21104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:42:52.570355 21104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:42:52.575484 21104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476718023131-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:42:52.576474 21104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:52.577450 21104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:52.585482 21112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:52.585482 21109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:52.585613 21104 server_base.cc:1034] running on GCE node
W20260927 02:42:52.585484 21110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:52.586344 21104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:52.587445 21104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:52.588620 21104 hybrid_clock.cc:648] HybridClock initialized: now 1790476972588593 us; error 36 us; skew 500 ppm
Sep 27 02:42:52 dist-test-slave-zv8z krb5kdc[21091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:52.595611 21104 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:42:52.596493 21104 webserver.cc:460] Webserver started at http://127.0.0.1:33709/ using document root <none> and password file <none>
I20260927 02:42:52.597003 21104 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:52.597059 21104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:52.597224 21104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:52.599210 21104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476718023131-392-0/minicluster-data/master-0/data/instance:
uuid: "eae9432113814ff7aee51456992df269"
format_stamp: "Formatted at 2026-09-27 02:42:52 on dist-test-slave-zv8z"
server_key: "bc9fcd5a4651e76c2b0348473d8349fa"
server_key_iv: "5e9854b429b94a40b7ce211b053d24fd"
server_key_version: "encryptionkey@0"
I20260927 02:42:52.599705 21104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476718023131-392-0/minicluster-data/master-0/wal/instance:
uuid: "eae9432113814ff7aee51456992df269"
format_stamp: "Formatted at 2026-09-27 02:42:52 on dist-test-slave-zv8z"
server_key: "bc9fcd5a4651e76c2b0348473d8349fa"
server_key_iv: "5e9854b429b94a40b7ce211b053d24fd"
server_key_version: "encryptionkey@0"
I20260927 02:42:52.603250 21104 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:42:52.605479 21119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:52.606762 21104 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:42:52.606859 21104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "eae9432113814ff7aee51456992df269"
format_stamp: "Formatted at 2026-09-27 02:42:52 on dist-test-slave-zv8z"
server_key: "bc9fcd5a4651e76c2b0348473d8349fa"
server_key_iv: "5e9854b429b94a40b7ce211b053d24fd"
server_key_version: "encryptionkey@0"
I20260927 02:42:52.606961 21104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:52.629735 21104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:52.633118 21104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:52.633297 21104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:52.639958 21104 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46069
I20260927 02:42:52.639968 21171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46069 every 8 connection(s)
I20260927 02:42:52.640887 21104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:42:52.643539 21172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:52.646751 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 21104
I20260927 02:42:52.646873 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790476718023131-392-0/minicluster-data/master-0/wal/instance
I20260927 02:42:52.647117 392 external_mini_cluster.cc:1428] Setting key 96b5e7706c7bcd460129626d17a963d0
I20260927 02:42:52.649107 21172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eae9432113814ff7aee51456992df269: Bootstrap starting.
I20260927 02:42:52.651778 21172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eae9432113814ff7aee51456992df269: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:52.652590 21172 log.cc:824] T 00000000000000000000000000000000 P eae9432113814ff7aee51456992df269: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:52.654529 21172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eae9432113814ff7aee51456992df269: No bootstrap required, opened a new log
I20260927 02:42:52.656787 21175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:52.648381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44265 (local address 127.0.0.1:46069)
0927 02:42:52.648786 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:52.648806 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:52.648847 (+ 41us) server_negotiation.cc:407] Connection header received
0927 02:42:52.649646 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:52.649669 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:52.650083 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:52.650485 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:52.650886 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:52.652362 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:52.653265 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:52.653730 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:52.653880 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:52.654675 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:52.654682 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:52.654697 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:42:52.655167 (+ 470us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:52.655243 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:42:52.655273 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:52.655367 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:52.655457 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 02:42:52.655654 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":94,"threads_started":1}
I20260927 02:42:52.658048 21172 raft_consensus.cc:348] T 00000000000000000000000000000000 P eae9432113814ff7aee51456992df269 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eae9432113814ff7aee51456992df269" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46069 } }
I20260927 02:42:52.658322 21172 raft_consensus.cc:374] T 00000000000000000000000000000000 P eae9432113814ff7aee51456992df269 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:52.658385 21172 raft_consensus.cc:729] T 00000000000000000000000000000000 P eae9432113814ff7aee51456992df269 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eae9432113814ff7aee51456992df269, State: Initialized, Role: FOLLOWER
I20260927 02:42:52.658825 21172 consensus_queue.cc:260] T 00000000000000000000000000000000 P eae9432113814ff7aee51456992df269 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eae9432113814ff7aee51456992df269" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46069 } }
I20260927 02:42:52.658967 21172 raft_consensus.cc:388] T 00000000000000000000000000000000 P eae9432113814ff7aee51456992df269 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:42:52.659013 21172 raft_consensus.cc:482] T 00000000000000000000000000000000 P eae9432113814ff7aee51456992df269 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:42:52.659102 21172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eae9432113814ff7aee51456992df269 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:52.659667 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:42:52.660123 21172 raft_consensus.cc:504] T 00000000000000000000000000000000 P eae9432113814ff7aee51456992df269 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eae9432113814ff7aee51456992df269" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46069 } }
I20260927 02:42:52.660477 21172 leader_election.cc:302] T 00000000000000000000000000000000 P eae9432113814ff7aee51456992df269 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eae9432113814ff7aee51456992df269; no voters:
I20260927 02:42:52.660820 21172 leader_election.cc:288] T 00000000000000000000000000000000 P eae9432113814ff7aee51456992df269 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:42:52.660930 21177 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eae9432113814ff7aee51456992df269 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:52.661181 21177 raft_consensus.cc:686] T 00000000000000000000000000000000 P eae9432113814ff7aee51456992df269 [term 1 LEADER]: Becoming Leader. State: Replica: eae9432113814ff7aee51456992df269, State: Running, Role: LEADER
I20260927 02:42:52.661505 21177 consensus_queue.cc:237] T 00000000000000000000000000000000 P eae9432113814ff7aee51456992df269 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eae9432113814ff7aee51456992df269" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46069 } }
I20260927 02:42:52.662068 21172 sys_catalog.cc:564] T 00000000000000000000000000000000 P eae9432113814ff7aee51456992df269 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:42:52.662858 21177 sys_catalog.cc:455] T 00000000000000000000000000000000 P eae9432113814ff7aee51456992df269 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eae9432113814ff7aee51456992df269. Latest consensus state: current_term: 1 leader_uuid: "eae9432113814ff7aee51456992df269" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eae9432113814ff7aee51456992df269" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46069 } } }
I20260927 02:42:52.663029 21177 sys_catalog.cc:458] T 00000000000000000000000000000000 P eae9432113814ff7aee51456992df269 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:52.663231 21182 sys_catalog.cc:455] T 00000000000000000000000000000000 P eae9432113814ff7aee51456992df269 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eae9432113814ff7aee51456992df269" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eae9432113814ff7aee51456992df269" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46069 } } }
I20260927 02:42:52.663316 21182 sys_catalog.cc:458] T 00000000000000000000000000000000 P eae9432113814ff7aee51456992df269 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:52.663452 21191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:42:52.666316 21175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:52.661733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34877 (local address 127.0.0.1:46069)
0927 02:42:52.661830 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:52.661835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:52.662096 (+ 261us) server_negotiation.cc:407] Connection header received
0927 02:42:52.662241 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:52.662244 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:52.662313 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:52.662394 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:52.662663 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:52.663424 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:52.664587 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:52.664886 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:52.665015 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:52.665831 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:52.665835 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:52.665838 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:42:52.665850 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:52.665880 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:42:52.665893 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:52.666006 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:52.666106 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 02:42:52.666161 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 02:42:52.666466 21191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:42:52.673242 21191 catalog_manager.cc:1347] Generated new cluster ID: d68304ec1dee4edbbf82362a080842cf
I20260927 02:42:52.673311 21191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:42:52.679591 21191 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:42:52.680545 21191 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:42:52.689597 21191 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eae9432113814ff7aee51456992df269: Generated new TSK 0
I20260927 02:42:52.690385 21191 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:42:52.702155 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 21104
2026-09-27T02:42:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1294 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476718023131-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:42:52 dist-test-slave-zv8z krb5kdc[21203](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:42:52 dist-test-slave-zv8z krb5kdc[21203](info): set up 2 sockets
Sep 27 02:42:52 dist-test-slave-zv8z krb5kdc[21203](info): commencing operation
krb5kdc: starting...
W20260927 02:42:53.650141 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476718023131-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476718023131-392-0/krb5kdc/test-user.keytab.
Sep 27 02:42:53 dist-test-slave-zv8z krb5kdc[21203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476973, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:42:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:42:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476718023131-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476718023131-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:42:53.762923 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39843
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476718023131-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:53.867424 21216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:53.867674 21216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:53.867714 21216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:53.871810 21216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:42:53.871869 21216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:53.871888 21216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:53.871904 21216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:42:53.871932 21216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:42:53.877190 21216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476718023131-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:42:53.878167 21216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:53.879158 21216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:53.887131 21224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:53.887123 21221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:53.887125 21222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:53.887423 21216 server_base.cc:1034] running on GCE node
I20260927 02:42:53.888016 21216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:53.889056 21216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:53.890234 21216 hybrid_clock.cc:648] HybridClock initialized: now 1790476973890207 us; error 40 us; skew 500 ppm
Sep 27 02:42:53 dist-test-slave-zv8z krb5kdc[21203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:53.892904 21216 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:42:53.893752 21216 webserver.cc:460] Webserver started at http://127.0.0.1:34019/ using document root <none> and password file <none>
I20260927 02:42:53.894307 21216 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:53.894375 21216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:53.894575 21216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:53.896526 21216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476718023131-392-0/minicluster-data/master-0/data/instance:
uuid: "7d9483aafa3243098d1741f42c8e14d0"
format_stamp: "Formatted at 2026-09-27 02:42:53 on dist-test-slave-zv8z"
server_key: "f990092e797254cc580c5eef4e5d0870"
server_key_iv: "aebb07f5bafdca204399aa5a4069d9e0"
server_key_version: "encryptionkey@0"
I20260927 02:42:53.897030 21216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476718023131-392-0/minicluster-data/master-0/wal/instance:
uuid: "7d9483aafa3243098d1741f42c8e14d0"
format_stamp: "Formatted at 2026-09-27 02:42:53 on dist-test-slave-zv8z"
server_key: "f990092e797254cc580c5eef4e5d0870"
server_key_iv: "aebb07f5bafdca204399aa5a4069d9e0"
server_key_version: "encryptionkey@0"
I20260927 02:42:53.900537 21216 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260927 02:42:53.902789 21231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:53.903970 21216 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:42:53.904100 21216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "7d9483aafa3243098d1741f42c8e14d0"
format_stamp: "Formatted at 2026-09-27 02:42:53 on dist-test-slave-zv8z"
server_key: "f990092e797254cc580c5eef4e5d0870"
server_key_iv: "aebb07f5bafdca204399aa5a4069d9e0"
server_key_version: "encryptionkey@0"
I20260927 02:42:53.904189 21216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:53.908542 21216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:53.911175 21216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:53.911329 21216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:53.917641 21216 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39843
I20260927 02:42:53.917714 21283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39843 every 8 connection(s)
I20260927 02:42:53.918550 21216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:42:53.921059 21284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:53.926066 21284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d9483aafa3243098d1741f42c8e14d0: Bootstrap starting.
I20260927 02:42:53.927928 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 21216
I20260927 02:42:53.928052 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790476718023131-392-0/minicluster-data/master-0/wal/instance
I20260927 02:42:53.928303 392 external_mini_cluster.cc:1428] Setting key d3ba230453587ee6722674c56477225a
I20260927 02:42:53.928404 21284 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d9483aafa3243098d1741f42c8e14d0: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:53.929221 21284 log.cc:824] T 00000000000000000000000000000000 P 7d9483aafa3243098d1741f42c8e14d0: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:53.930980 21284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d9483aafa3243098d1741f42c8e14d0: No bootstrap required, opened a new log
I20260927 02:42:53.932760 21287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:53.929268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39929 (local address 127.0.0.1:39843)
0927 02:42:53.929663 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:53.929682 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:53.929721 (+ 39us) server_negotiation.cc:407] Connection header received
0927 02:42:53.930363 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:53.930380 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:53.930646 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:53.930906 (+ 260us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:53.931655 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:53.931659 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:53.931671 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:42:53.931681 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:53.931729 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:42:53.931746 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:53.931823 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:53.931942 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 02:42:53.932095 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":121,"threads_started":1}
I20260927 02:42:53.934561 21284 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d9483aafa3243098d1741f42c8e14d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d9483aafa3243098d1741f42c8e14d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39843 } }
I20260927 02:42:53.934804 21284 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d9483aafa3243098d1741f42c8e14d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:53.934861 21284 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d9483aafa3243098d1741f42c8e14d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d9483aafa3243098d1741f42c8e14d0, State: Initialized, Role: FOLLOWER
I20260927 02:42:53.935182 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:42:53.935273 21284 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d9483aafa3243098d1741f42c8e14d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d9483aafa3243098d1741f42c8e14d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39843 } }
I20260927 02:42:53.935406 21284 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d9483aafa3243098d1741f42c8e14d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:42:53.935465 21284 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d9483aafa3243098d1741f42c8e14d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:42:53.935554 21284 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d9483aafa3243098d1741f42c8e14d0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:53.936347 21284 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d9483aafa3243098d1741f42c8e14d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d9483aafa3243098d1741f42c8e14d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39843 } }
I20260927 02:42:53.936655 21284 leader_election.cc:302] T 00000000000000000000000000000000 P 7d9483aafa3243098d1741f42c8e14d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d9483aafa3243098d1741f42c8e14d0; no voters:
I20260927 02:42:53.936919 21284 leader_election.cc:288] T 00000000000000000000000000000000 P 7d9483aafa3243098d1741f42c8e14d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:42:53.937012 21293 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d9483aafa3243098d1741f42c8e14d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:53.937208 21293 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d9483aafa3243098d1741f42c8e14d0 [term 1 LEADER]: Becoming Leader. State: Replica: 7d9483aafa3243098d1741f42c8e14d0, State: Running, Role: LEADER
I20260927 02:42:53.937552 21293 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d9483aafa3243098d1741f42c8e14d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d9483aafa3243098d1741f42c8e14d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39843 } }
I20260927 02:42:53.938020 21284 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d9483aafa3243098d1741f42c8e14d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:42:53.939123 21296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d9483aafa3243098d1741f42c8e14d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d9483aafa3243098d1741f42c8e14d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d9483aafa3243098d1741f42c8e14d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39843 } } }
I20260927 02:42:53.939237 21296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d9483aafa3243098d1741f42c8e14d0 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:53.939447 21296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d9483aafa3243098d1741f42c8e14d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d9483aafa3243098d1741f42c8e14d0. Latest consensus state: current_term: 1 leader_uuid: "7d9483aafa3243098d1741f42c8e14d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d9483aafa3243098d1741f42c8e14d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39843 } } }
I20260927 02:42:53.939520 21296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d9483aafa3243098d1741f42c8e14d0 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:53.939733 21287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:53.937327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50999 (local address 127.0.0.1:39843)
0927 02:42:53.937418 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:53.937422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:53.937637 (+ 215us) server_negotiation.cc:407] Connection header received
0927 02:42:53.937805 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:53.937808 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:53.937838 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:53.937883 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:53.938809 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:53.938812 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:53.938815 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:42:53.938828 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:53.938854 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:42:53.938897 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:53.938958 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:53.939147 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 02:42:53.939185 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260927 02:42:53.942625 21309 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7d9483aafa3243098d1741f42c8e14d0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:42:53.942687 21309 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:42:53.942750 21303 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:42:53.943405 21303 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:42:53.948338 21303 catalog_manager.cc:1347] Generated new cluster ID: 2082a97d16f545fc87517d980d9ad06b
I20260927 02:42:53.948392 21303 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:42:53.961563 21303 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:42:53.961871 21303 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:42:53.974426 21303 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d9483aafa3243098d1741f42c8e14d0: Generated new TSK 0
I20260927 02:42:53.975207 21303 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:42:53.983734 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 21216
2026-09-27T02:42:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1281 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476718023131-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:42:54 dist-test-slave-zv8z krb5kdc[21315](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:42:54 dist-test-slave-zv8z krb5kdc[21315](info): set up 2 sockets
Sep 27 02:42:54 dist-test-slave-zv8z krb5kdc[21315](info): commencing operation
krb5kdc: starting...
W20260927 02:42:54.922154 392 mini_kdc.cc:121] Time spent starting KDC: real 0.920s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476718023131-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476718023131-392-0/krb5kdc/test-user.keytab.
Sep 27 02:42:54 dist-test-slave-zv8z krb5kdc[21315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:42:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:42:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476718023131-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476718023131-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:42:55.033119 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34713
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476718023131-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34713
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:55.140844 21328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:55.141124 21328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:55.141183 21328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:55.145233 21328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:42:55.145304 21328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:55.145332 21328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:55.145352 21328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:42:55.145373 21328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:42:55.150813 21328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34713
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476718023131-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:42:55.151890 21328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:55.152909 21328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:55.161350 21333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:55.161415 21328 server_base.cc:1034] running on GCE node
W20260927 02:42:55.161444 21334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:55.161357 21336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:55.162111 21328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:55.163211 21328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:55.164377 21328 hybrid_clock.cc:648] HybridClock initialized: now 1790476975164357 us; error 32 us; skew 500 ppm
Sep 27 02:42:55 dist-test-slave-zv8z krb5kdc[21315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:55.167088 21328 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:42:55.167958 21328 webserver.cc:460] Webserver started at http://127.0.0.1:42645/ using document root <none> and password file <none>
I20260927 02:42:55.168480 21328 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:55.168536 21328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:55.168722 21328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:55.170746 21328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476718023131-392-0/minicluster-data/master-0/data/instance:
uuid: "d4f5d6de348d4d39868d4be32e45316f"
format_stamp: "Formatted at 2026-09-27 02:42:55 on dist-test-slave-zv8z"
server_key: "db816e88eaae2ceafed180c88432d550"
server_key_iv: "77fc9dea55d755c37e91d7d412cb9f13"
server_key_version: "encryptionkey@0"
I20260927 02:42:55.171262 21328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476718023131-392-0/minicluster-data/master-0/wal/instance:
uuid: "d4f5d6de348d4d39868d4be32e45316f"
format_stamp: "Formatted at 2026-09-27 02:42:55 on dist-test-slave-zv8z"
server_key: "db816e88eaae2ceafed180c88432d550"
server_key_iv: "77fc9dea55d755c37e91d7d412cb9f13"
server_key_version: "encryptionkey@0"
I20260927 02:42:55.174736 21328 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 02:42:55.176927 21343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:55.178083 21328 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:42:55.178197 21328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "d4f5d6de348d4d39868d4be32e45316f"
format_stamp: "Formatted at 2026-09-27 02:42:55 on dist-test-slave-zv8z"
server_key: "db816e88eaae2ceafed180c88432d550"
server_key_iv: "77fc9dea55d755c37e91d7d412cb9f13"
server_key_version: "encryptionkey@0"
I20260927 02:42:55.178313 21328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:55.201089 21328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:55.208312 21328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:55.208559 21328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:55.217659 21328 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34713
I20260927 02:42:55.217701 21395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34713 every 8 connection(s)
I20260927 02:42:55.218947 21328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:42:55.222442 21396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:55.228878 21396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4f5d6de348d4d39868d4be32e45316f: Bootstrap starting.
I20260927 02:42:55.229161 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 21328
I20260927 02:42:55.229319 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790476718023131-392-0/minicluster-data/master-0/wal/instance
I20260927 02:42:55.229852 392 external_mini_cluster.cc:1428] Setting key f1ab44a2c08406c0d4fbaae2ae18ff7a
I20260927 02:42:55.231933 21396 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4f5d6de348d4d39868d4be32e45316f: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:55.232723 21396 log.cc:824] T 00000000000000000000000000000000 P d4f5d6de348d4d39868d4be32e45316f: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:55.235162 21396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4f5d6de348d4d39868d4be32e45316f: No bootstrap required, opened a new log
I20260927 02:42:55.239084 21396 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4f5d6de348d4d39868d4be32e45316f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f5d6de348d4d39868d4be32e45316f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34713 } }
I20260927 02:42:55.239404 21396 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4f5d6de348d4d39868d4be32e45316f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
Sep 27 02:42:55 dist-test-slave-zv8z krb5kdc[21315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 02:42:55.239462 21396 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4f5d6de348d4d39868d4be32e45316f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4f5d6de348d4d39868d4be32e45316f, State: Initialized, Role: FOLLOWER
I20260927 02:42:55.239950 21396 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4f5d6de348d4d39868d4be32e45316f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f5d6de348d4d39868d4be32e45316f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34713 } }
I20260927 02:42:55.240099 21396 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4f5d6de348d4d39868d4be32e45316f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:42:55.240208 21396 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4f5d6de348d4d39868d4be32e45316f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:42:55.240306 21396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4f5d6de348d4d39868d4be32e45316f [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:55.241333 21396 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4f5d6de348d4d39868d4be32e45316f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f5d6de348d4d39868d4be32e45316f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34713 } }
I20260927 02:42:55.241696 21396 leader_election.cc:302] T 00000000000000000000000000000000 P d4f5d6de348d4d39868d4be32e45316f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4f5d6de348d4d39868d4be32e45316f; no voters:
I20260927 02:42:55.242035 21396 leader_election.cc:288] T 00000000000000000000000000000000 P d4f5d6de348d4d39868d4be32e45316f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:42:55.242136 21401 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4f5d6de348d4d39868d4be32e45316f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:55.242398 21401 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4f5d6de348d4d39868d4be32e45316f [term 1 LEADER]: Becoming Leader. State: Replica: d4f5d6de348d4d39868d4be32e45316f, State: Running, Role: LEADER
I20260927 02:42:55.242692 21401 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4f5d6de348d4d39868d4be32e45316f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f5d6de348d4d39868d4be32e45316f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34713 } }
I20260927 02:42:55.243424 21396 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4f5d6de348d4d39868d4be32e45316f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:42:55.243860 21401 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4f5d6de348d4d39868d4be32e45316f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4f5d6de348d4d39868d4be32e45316f. Latest consensus state: current_term: 1 leader_uuid: "d4f5d6de348d4d39868d4be32e45316f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f5d6de348d4d39868d4be32e45316f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34713 } } }
I20260927 02:42:55.243973 21401 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4f5d6de348d4d39868d4be32e45316f [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:55.244360 21405 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:42:55.245961 21399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:55.231566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:41793 (local address 127.0.0.1:34713)
0927 02:42:55.232182 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:55.232197 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:55.232230 (+ 33us) server_negotiation.cc:407] Connection header received
0927 02:42:55.233019 (+ 789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:55.233039 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:55.233445 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:55.233818 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:55.234698 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:55.235802 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:55.236644 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:55.237146 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:55.237288 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:55.240053 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:55.240058 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:55.240073 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:55.240107 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:55.242094 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:55.242741 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:55.242747 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:55.242753 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:55.242889 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:55.243246 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:55.243249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:55.243251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:55.243735 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:55.243958 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:55.244815 (+ 857us) server_negotiation.cc:299] Negotiation successful
0927 02:42:55.245029 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":123,"threads_started":1}
I20260927 02:42:55.248099 21405 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:42:55.249143 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:42:55.250739 21402 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4f5d6de348d4d39868d4be32e45316f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4f5d6de348d4d39868d4be32e45316f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f5d6de348d4d39868d4be32e45316f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34713 } } }
I20260927 02:42:55.250864 21402 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4f5d6de348d4d39868d4be32e45316f [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:55.254724 21405 catalog_manager.cc:1347] Generated new cluster ID: 2013373f842b4629a9e940d850b1c4c5
I20260927 02:42:55.254783 21405 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:42:55.263796 21399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:55.251535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:37537 (local address 127.0.0.1:34713)
0927 02:42:55.251656 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:55.251660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:55.251765 (+ 105us) server_negotiation.cc:407] Connection header received
0927 02:42:55.251868 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:55.251871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:55.251913 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:55.251973 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:55.252587 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:55.253216 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:55.258349 (+ 5133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:55.258705 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:55.258827 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:55.260521 (+ 1694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:55.260525 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:55.260528 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:55.260550 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:55.262066 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:55.262616 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:55.262619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:55.262621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:55.262705 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:55.262985 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:55.262988 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:55.262990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:55.263175 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:55.263301 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:55.263547 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 02:42:55.263638 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 02:42:55.272027 21405 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:42:55.272971 21405 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:42:55.282333 21405 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4f5d6de348d4d39868d4be32e45316f: Generated new TSK 0
I20260927 02:42:55.283600 21405 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:42:55.290637 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 21328
W20260927 02:42:55.296510 21313 connection.cc:535] client connection to 127.0.0.1:34713 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34713): Connection reset by peer (error 104)
2026-09-27T02:42:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1308 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476718023131-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:42:55 dist-test-slave-zv8z krb5kdc[21427](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:42:55 dist-test-slave-zv8z krb5kdc[21427](info): set up 2 sockets
Sep 27 02:42:55 dist-test-slave-zv8z krb5kdc[21427](info): commencing operation
krb5kdc: starting...
W20260927 02:42:56.230109 392 mini_kdc.cc:121] Time spent starting KDC: real 0.920s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476718023131-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476718023131-392-0/krb5kdc/test-user.keytab.
Sep 27 02:42:56 dist-test-slave-zv8z krb5kdc[21427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:42:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:42:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476718023131-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476718023131-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:42:56.343667 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36361
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476718023131-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36361
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:56.451010 21440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:56.451251 21440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:56.451295 21440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:56.455245 21440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:42:56.455300 21440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:56.455317 21440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:56.455333 21440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:42:56.455346 21440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:42:56.460538 21440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476718023131-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:42:56.461485 21440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:56.462455 21440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:56.470368 21448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:56.470381 21446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:56.470379 21445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:56.470551 21440 server_base.cc:1034] running on GCE node
I20260927 02:42:56.471047 21440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:56.472064 21440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:56.473220 21440 hybrid_clock.cc:648] HybridClock initialized: now 1790476976473181 us; error 58 us; skew 500 ppm
Sep 27 02:42:56 dist-test-slave-zv8z krb5kdc[21427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476976, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:56.475760 21440 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:42:56.476558 21440 webserver.cc:460] Webserver started at http://127.0.0.1:39419/ using document root <none> and password file <none>
I20260927 02:42:56.477046 21440 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:56.477099 21440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:56.477262 21440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:56.479202 21440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476718023131-392-0/minicluster-data/master-0/data/instance:
uuid: "17e05da99a80419381c72cf8eafb9f11"
format_stamp: "Formatted at 2026-09-27 02:42:56 on dist-test-slave-zv8z"
server_key: "7491dff51da1d43ffda848ab88f0376d"
server_key_iv: "fb09f86c177b00ba3416aaff1513052b"
server_key_version: "encryptionkey@0"
I20260927 02:42:56.479665 21440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476718023131-392-0/minicluster-data/master-0/wal/instance:
uuid: "17e05da99a80419381c72cf8eafb9f11"
format_stamp: "Formatted at 2026-09-27 02:42:56 on dist-test-slave-zv8z"
server_key: "7491dff51da1d43ffda848ab88f0376d"
server_key_iv: "fb09f86c177b00ba3416aaff1513052b"
server_key_version: "encryptionkey@0"
I20260927 02:42:56.483062 21440 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:42:56.485301 21455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:56.486502 21440 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:42:56.486665 21440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "17e05da99a80419381c72cf8eafb9f11"
format_stamp: "Formatted at 2026-09-27 02:42:56 on dist-test-slave-zv8z"
server_key: "7491dff51da1d43ffda848ab88f0376d"
server_key_iv: "fb09f86c177b00ba3416aaff1513052b"
server_key_version: "encryptionkey@0"
I20260927 02:42:56.486764 21440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:56.514604 21440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:56.517767 21440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:56.517935 21440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:56.524406 21440 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36361
I20260927 02:42:56.524451 21507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36361 every 8 connection(s)
I20260927 02:42:56.525313 21440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:42:56.527917 21508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:56.529564 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 21440
I20260927 02:42:56.529708 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790476718023131-392-0/minicluster-data/master-0/wal/instance
I20260927 02:42:56.529971 392 external_mini_cluster.cc:1428] Setting key 5ebbf5df378bfe15d7826281a2da1d47
I20260927 02:42:56.533968 21508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17e05da99a80419381c72cf8eafb9f11: Bootstrap starting.
I20260927 02:42:56.536422 21508 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17e05da99a80419381c72cf8eafb9f11: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:56.537148 21508 log.cc:824] T 00000000000000000000000000000000 P 17e05da99a80419381c72cf8eafb9f11: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:56.538370 21511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:56.531258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:40193 (local address 127.0.0.1:36361)
0927 02:42:56.531639 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:56.531657 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:56.531691 (+ 34us) server_negotiation.cc:407] Connection header received
0927 02:42:56.532296 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:56.532312 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:56.532633 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:56.532899 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:56.533210 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:56.534160 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:56.535318 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:56.535774 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:56.535885 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:56.536714 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:56.536719 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:56.536731 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:42:56.537077 (+ 346us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:56.537146 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:42:56.537169 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:56.537285 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:56.537455 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 02:42:56.537583 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":128,"threads_started":1}
I20260927 02:42:56.538982 21508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17e05da99a80419381c72cf8eafb9f11: No bootstrap required, opened a new log
I20260927 02:42:56.541044 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:42:56.542829 21508 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17e05da99a80419381c72cf8eafb9f11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17e05da99a80419381c72cf8eafb9f11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36361 } }
I20260927 02:42:56.543102 21508 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17e05da99a80419381c72cf8eafb9f11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:56.543169 21508 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17e05da99a80419381c72cf8eafb9f11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17e05da99a80419381c72cf8eafb9f11, State: Initialized, Role: FOLLOWER
I20260927 02:42:56.543581 21508 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17e05da99a80419381c72cf8eafb9f11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17e05da99a80419381c72cf8eafb9f11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36361 } }
I20260927 02:42:56.543705 21508 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17e05da99a80419381c72cf8eafb9f11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:42:56.543766 21508 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17e05da99a80419381c72cf8eafb9f11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:42:56.543854 21508 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17e05da99a80419381c72cf8eafb9f11 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:56.544674 21508 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17e05da99a80419381c72cf8eafb9f11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17e05da99a80419381c72cf8eafb9f11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36361 } }
I20260927 02:42:56.544970 21508 leader_election.cc:302] T 00000000000000000000000000000000 P 17e05da99a80419381c72cf8eafb9f11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17e05da99a80419381c72cf8eafb9f11; no voters:
I20260927 02:42:56.545238 21508 leader_election.cc:288] T 00000000000000000000000000000000 P 17e05da99a80419381c72cf8eafb9f11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:42:56.545317 21518 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17e05da99a80419381c72cf8eafb9f11 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:56.545482 21518 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17e05da99a80419381c72cf8eafb9f11 [term 1 LEADER]: Becoming Leader. State: Replica: 17e05da99a80419381c72cf8eafb9f11, State: Running, Role: LEADER
I20260927 02:42:56.545758 21518 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17e05da99a80419381c72cf8eafb9f11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17e05da99a80419381c72cf8eafb9f11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36361 } }
I20260927 02:42:56.546456 21508 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17e05da99a80419381c72cf8eafb9f11 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:42:56.546864 21511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:56.542852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:59809 (local address 127.0.0.1:36361)
0927 02:42:56.542944 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:56.542948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:56.543118 (+ 170us) server_negotiation.cc:407] Connection header received
0927 02:42:56.543239 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:56.543242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:56.543282 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:56.543343 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:56.543574 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:56.544200 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:56.545008 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:56.545338 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:56.545429 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:56.546324 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:56.546327 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:56.546330 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:42:56.546341 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:56.546364 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:42:56.546374 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:56.546437 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:56.546717 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 02:42:56.546749 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 02:42:56.546895 21518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17e05da99a80419381c72cf8eafb9f11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17e05da99a80419381c72cf8eafb9f11. Latest consensus state: current_term: 1 leader_uuid: "17e05da99a80419381c72cf8eafb9f11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17e05da99a80419381c72cf8eafb9f11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36361 } } }
I20260927 02:42:56.547001 21518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17e05da99a80419381c72cf8eafb9f11 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:56.547469 21519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17e05da99a80419381c72cf8eafb9f11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17e05da99a80419381c72cf8eafb9f11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17e05da99a80419381c72cf8eafb9f11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36361 } } }
I20260927 02:42:56.547559 21519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17e05da99a80419381c72cf8eafb9f11 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:56.547927 21527 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:42:56.550913 21527 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:42:56.556419 21527 catalog_manager.cc:1347] Generated new cluster ID: 213b0b66138f44fd866e7fc0d7c5bf1e
I20260927 02:42:56.556473 21527 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:42:56.565379 21527 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:42:56.566339 21527 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:42:56.570271 21527 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17e05da99a80419381c72cf8eafb9f11: Generated new TSK 0
I20260927 02:42:56.570904 21527 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:42:56.577538 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 21440
W20260927 02:42:56.582278 21425 connection.cc:535] client connection to 127.0.0.1:36361 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36361): Connection reset by peer (error 104)
2026-09-27T02:42:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1285 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476718023131-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:42:56 dist-test-slave-zv8z krb5kdc[21539](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:42:56 dist-test-slave-zv8z krb5kdc[21539](info): set up 2 sockets
Sep 27 02:42:56 dist-test-slave-zv8z krb5kdc[21539](info): commencing operation
krb5kdc: starting...
W20260927 02:42:57.522188 392 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476718023131-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476718023131-392-0/krb5kdc/test-user.keytab.
Sep 27 02:42:57 dist-test-slave-zv8z krb5kdc[21539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:42:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:42:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476718023131-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476718023131-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:42:57.636714 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45613
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476718023131-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:57.742146 21552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:57.742426 21552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:57.742466 21552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:57.746367 21552 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:42:57.746419 21552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:57.746438 21552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:57.746452 21552 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:42:57.746465 21552 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:42:57.751657 21552 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476718023131-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:42:57.752614 21552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:57.753564 21552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:57.761459 21558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:57.761503 21557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:57.761577 21552 server_base.cc:1034] running on GCE node
W20260927 02:42:57.761495 21560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:57.762115 21552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:57.763190 21552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:57.764424 21552 hybrid_clock.cc:648] HybridClock initialized: now 1790476977764400 us; error 37 us; skew 500 ppm
Sep 27 02:42:57 dist-test-slave-zv8z krb5kdc[21539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:57.770880 21552 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:42:57.771754 21552 webserver.cc:460] Webserver started at http://127.0.0.1:45545/ using document root <none> and password file <none>
I20260927 02:42:57.772279 21552 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:57.772339 21552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:57.772516 21552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:57.774524 21552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476718023131-392-0/minicluster-data/master-0/data/instance:
uuid: "2886972fadb142ad97c6ae1e5ac8952d"
format_stamp: "Formatted at 2026-09-27 02:42:57 on dist-test-slave-zv8z"
server_key: "58c09efa0af442e0c2a01c88a6ce21c5"
server_key_iv: "49bfcd74de1b799216687655bf625019"
server_key_version: "encryptionkey@0"
I20260927 02:42:57.775048 21552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476718023131-392-0/minicluster-data/master-0/wal/instance:
uuid: "2886972fadb142ad97c6ae1e5ac8952d"
format_stamp: "Formatted at 2026-09-27 02:42:57 on dist-test-slave-zv8z"
server_key: "58c09efa0af442e0c2a01c88a6ce21c5"
server_key_iv: "49bfcd74de1b799216687655bf625019"
server_key_version: "encryptionkey@0"
I20260927 02:42:57.778566 21552 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:42:57.780787 21567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:57.781994 21552 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:42:57.782131 21552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "2886972fadb142ad97c6ae1e5ac8952d"
format_stamp: "Formatted at 2026-09-27 02:42:57 on dist-test-slave-zv8z"
server_key: "58c09efa0af442e0c2a01c88a6ce21c5"
server_key_iv: "49bfcd74de1b799216687655bf625019"
server_key_version: "encryptionkey@0"
I20260927 02:42:57.782222 21552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:57.786392 21552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:57.789040 21552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:57.789196 21552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:57.795608 21552 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45613
I20260927 02:42:57.795650 21619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45613 every 8 connection(s)
I20260927 02:42:57.796510 21552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:42:57.799140 21620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:57.801888 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 21552
I20260927 02:42:57.801995 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790476718023131-392-0/minicluster-data/master-0/wal/instance
I20260927 02:42:57.802232 392 external_mini_cluster.cc:1428] Setting key 72eab4d020de68cae88a36a28ce40bef
I20260927 02:42:57.804847 21620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2886972fadb142ad97c6ae1e5ac8952d: Bootstrap starting.
I20260927 02:42:57.807483 21620 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2886972fadb142ad97c6ae1e5ac8952d: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:57.807749 21623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:57.803599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:45547 (local address 127.0.0.1:45613)
0927 02:42:57.803997 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:57.804016 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:57.804058 (+ 42us) server_negotiation.cc:407] Connection header received
0927 02:42:57.804861 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:57.804885 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:57.805250 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:57.805603 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:57.806330 (+ 727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:57.806335 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:57.806350 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:42:57.806361 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:57.806423 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:42:57.806448 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:57.806572 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:57.806692 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 02:42:57.806844 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":98,"threads_started":1}
I20260927 02:42:57.808326 21620 log.cc:824] T 00000000000000000000000000000000 P 2886972fadb142ad97c6ae1e5ac8952d: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:57.810321 21620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2886972fadb142ad97c6ae1e5ac8952d: No bootstrap required, opened a new log
I20260927 02:42:57.810499 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:42:57.813849 21623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:57.812364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:58043 (local address 127.0.0.1:45613)
0927 02:42:57.812459 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:57.812462 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:57.812673 (+ 211us) server_negotiation.cc:407] Connection header received
0927 02:42:57.812786 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:57.812788 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:57.812815 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:57.812874 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:57.813519 (+ 645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:57.813521 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:57.813523 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:42:57.813532 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:57.813554 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:42:57.813563 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:57.813633 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:57.813717 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 02:42:57.813748 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 02:42:57.814153 21620 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2886972fadb142ad97c6ae1e5ac8952d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2886972fadb142ad97c6ae1e5ac8952d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45613 } }
I20260927 02:42:57.814435 21620 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2886972fadb142ad97c6ae1e5ac8952d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:57.814517 21620 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2886972fadb142ad97c6ae1e5ac8952d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2886972fadb142ad97c6ae1e5ac8952d, State: Initialized, Role: FOLLOWER
I20260927 02:42:57.814966 21620 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2886972fadb142ad97c6ae1e5ac8952d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2886972fadb142ad97c6ae1e5ac8952d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45613 } }
I20260927 02:42:57.815094 21620 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2886972fadb142ad97c6ae1e5ac8952d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:42:57.815140 21620 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2886972fadb142ad97c6ae1e5ac8952d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:42:57.815209 21620 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2886972fadb142ad97c6ae1e5ac8952d [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:57.815980 21620 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2886972fadb142ad97c6ae1e5ac8952d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2886972fadb142ad97c6ae1e5ac8952d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45613 } }
I20260927 02:42:57.816260 21620 leader_election.cc:302] T 00000000000000000000000000000000 P 2886972fadb142ad97c6ae1e5ac8952d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2886972fadb142ad97c6ae1e5ac8952d; no voters:
I20260927 02:42:57.816478 21620 leader_election.cc:288] T 00000000000000000000000000000000 P 2886972fadb142ad97c6ae1e5ac8952d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:42:57.816579 21630 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2886972fadb142ad97c6ae1e5ac8952d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:57.816795 21630 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2886972fadb142ad97c6ae1e5ac8952d [term 1 LEADER]: Becoming Leader. State: Replica: 2886972fadb142ad97c6ae1e5ac8952d, State: Running, Role: LEADER
I20260927 02:42:57.817117 21630 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2886972fadb142ad97c6ae1e5ac8952d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2886972fadb142ad97c6ae1e5ac8952d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45613 } }
I20260927 02:42:57.817538 21620 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2886972fadb142ad97c6ae1e5ac8952d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:42:57.818533 21632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2886972fadb142ad97c6ae1e5ac8952d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2886972fadb142ad97c6ae1e5ac8952d. Latest consensus state: current_term: 1 leader_uuid: "2886972fadb142ad97c6ae1e5ac8952d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2886972fadb142ad97c6ae1e5ac8952d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45613 } } }
I20260927 02:42:57.818806 21631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2886972fadb142ad97c6ae1e5ac8952d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2886972fadb142ad97c6ae1e5ac8952d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2886972fadb142ad97c6ae1e5ac8952d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45613 } } }
I20260927 02:42:57.818918 21632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2886972fadb142ad97c6ae1e5ac8952d [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:57.819010 21631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2886972fadb142ad97c6ae1e5ac8952d [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:57.819365 21643 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:42:57.821987 21643 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:42:57.826867 21643 catalog_manager.cc:1347] Generated new cluster ID: c21b1a9f6efb47bfb79b22bcb4f53a9a
I20260927 02:42:57.826926 21643 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:42:57.846868 21643 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:42:57.847169 21643 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:42:57.852061 21643 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2886972fadb142ad97c6ae1e5ac8952d: Generated new TSK 0
I20260927 02:42:57.852692 21643 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:42:57.855068 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 21552
2026-09-27T02:42:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1277 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476718023131-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:42:57 dist-test-slave-zv8z krb5kdc[21651](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:42:57 dist-test-slave-zv8z krb5kdc[21651](info): set up 2 sockets
Sep 27 02:42:57 dist-test-slave-zv8z krb5kdc[21651](info): commencing operation
krb5kdc: starting...
W20260927 02:42:58.798280 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476718023131-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476718023131-392-0/krb5kdc/test-user.keytab.
Sep 27 02:42:58 dist-test-slave-zv8z krb5kdc[21651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:42:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:42:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476718023131-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476718023131-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:42:58.910066 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33251
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476718023131-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:42:59.017361 21664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:42:59.017627 21664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:42:59.017668 21664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:42:59.021656 21664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:42:59.021726 21664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:42:59.021744 21664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:42:59.021759 21664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:42:59.021772 21664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:42:59.027144 21664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33251
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476718023131-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:42:59.028146 21664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:42:59.029096 21664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:42:59.037364 21672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:42:59.037395 21670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:59.037449 21664 server_base.cc:1034] running on GCE node
W20260927 02:42:59.037395 21669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:42:59.038159 21664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:42:59.039234 21664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:42:59.040400 21664 hybrid_clock.cc:648] HybridClock initialized: now 1790476979040384 us; error 29 us; skew 500 ppm
Sep 27 02:42:59 dist-test-slave-zv8z krb5kdc[21651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:42:59.042949 21664 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:42:59.043789 21664 webserver.cc:460] Webserver started at http://127.0.0.1:44445/ using document root <none> and password file <none>
I20260927 02:42:59.044305 21664 fs_manager.cc:359] Metadata directory not provided
I20260927 02:42:59.044378 21664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:42:59.044549 21664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:42:59.046545 21664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476718023131-392-0/minicluster-data/master-0/data/instance:
uuid: "37356d01fa5d4cdf9fad29a6245133b2"
format_stamp: "Formatted at 2026-09-27 02:42:59 on dist-test-slave-zv8z"
server_key: "b1fae73f7da477de3bf14d14a9f8bc2e"
server_key_iv: "291b479ef300c2aaffad64127f589a41"
server_key_version: "encryptionkey@0"
I20260927 02:42:59.047051 21664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476718023131-392-0/minicluster-data/master-0/wal/instance:
uuid: "37356d01fa5d4cdf9fad29a6245133b2"
format_stamp: "Formatted at 2026-09-27 02:42:59 on dist-test-slave-zv8z"
server_key: "b1fae73f7da477de3bf14d14a9f8bc2e"
server_key_iv: "291b479ef300c2aaffad64127f589a41"
server_key_version: "encryptionkey@0"
I20260927 02:42:59.050586 21664 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 02:42:59.052919 21679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:42:59.054177 21664 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:42:59.054338 21664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "37356d01fa5d4cdf9fad29a6245133b2"
format_stamp: "Formatted at 2026-09-27 02:42:59 on dist-test-slave-zv8z"
server_key: "b1fae73f7da477de3bf14d14a9f8bc2e"
server_key_iv: "291b479ef300c2aaffad64127f589a41"
server_key_version: "encryptionkey@0"
I20260927 02:42:59.054453 21664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:42:59.066148 21664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:42:59.069242 21664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:42:59.069413 21664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:42:59.076223 21664 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33251
I20260927 02:42:59.076237 21731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33251 every 8 connection(s)
I20260927 02:42:59.077170 21664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:42:59.079852 21732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:42:59.085232 21732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37356d01fa5d4cdf9fad29a6245133b2: Bootstrap starting.
I20260927 02:42:59.085635 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 21664
I20260927 02:42:59.085765 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790476718023131-392-0/minicluster-data/master-0/wal/instance
I20260927 02:42:59.085999 392 external_mini_cluster.cc:1428] Setting key 9bd0cd15578e5df411db673e83d29604
I20260927 02:42:59.087899 21732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37356d01fa5d4cdf9fad29a6245133b2: Neither blocks nor log segments found. Creating new log.
I20260927 02:42:59.088611 21732 log.cc:824] T 00000000000000000000000000000000 P 37356d01fa5d4cdf9fad29a6245133b2: Log is configured to *not* fsync() on all Append() calls
I20260927 02:42:59.090718 21732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37356d01fa5d4cdf9fad29a6245133b2: No bootstrap required, opened a new log
Sep 27 02:42:59 dist-test-slave-zv8z krb5kdc[21651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 02:42:59.095005 21732 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37356d01fa5d4cdf9fad29a6245133b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37356d01fa5d4cdf9fad29a6245133b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33251 } }
I20260927 02:42:59.095228 21732 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37356d01fa5d4cdf9fad29a6245133b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:42:59.095283 21732 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37356d01fa5d4cdf9fad29a6245133b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37356d01fa5d4cdf9fad29a6245133b2, State: Initialized, Role: FOLLOWER
I20260927 02:42:59.095698 21732 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37356d01fa5d4cdf9fad29a6245133b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37356d01fa5d4cdf9fad29a6245133b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33251 } }
I20260927 02:42:59.095820 21732 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37356d01fa5d4cdf9fad29a6245133b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:42:59.095882 21732 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37356d01fa5d4cdf9fad29a6245133b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:42:59.095988 21732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37356d01fa5d4cdf9fad29a6245133b2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:42:59.097011 21732 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37356d01fa5d4cdf9fad29a6245133b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37356d01fa5d4cdf9fad29a6245133b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33251 } }
I20260927 02:42:59.097330 21732 leader_election.cc:302] T 00000000000000000000000000000000 P 37356d01fa5d4cdf9fad29a6245133b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37356d01fa5d4cdf9fad29a6245133b2; no voters:
I20260927 02:42:59.097622 21732 leader_election.cc:288] T 00000000000000000000000000000000 P 37356d01fa5d4cdf9fad29a6245133b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:42:59.097754 21737 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37356d01fa5d4cdf9fad29a6245133b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:42:59.097978 21737 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37356d01fa5d4cdf9fad29a6245133b2 [term 1 LEADER]: Becoming Leader. State: Replica: 37356d01fa5d4cdf9fad29a6245133b2, State: Running, Role: LEADER
I20260927 02:42:59.098395 21737 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37356d01fa5d4cdf9fad29a6245133b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37356d01fa5d4cdf9fad29a6245133b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33251 } }
I20260927 02:42:59.098949 21732 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37356d01fa5d4cdf9fad29a6245133b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:42:59.100028 21739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37356d01fa5d4cdf9fad29a6245133b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37356d01fa5d4cdf9fad29a6245133b2. Latest consensus state: current_term: 1 leader_uuid: "37356d01fa5d4cdf9fad29a6245133b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37356d01fa5d4cdf9fad29a6245133b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33251 } } }
I20260927 02:42:59.100157 21739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37356d01fa5d4cdf9fad29a6245133b2 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:59.100494 21738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37356d01fa5d4cdf9fad29a6245133b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37356d01fa5d4cdf9fad29a6245133b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37356d01fa5d4cdf9fad29a6245133b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33251 } } }
I20260927 02:42:59.100651 21738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37356d01fa5d4cdf9fad29a6245133b2 [sys.catalog]: This master's current role is: LEADER
I20260927 02:42:59.102144 21735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:59.087270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:59681 (local address 127.0.0.1:33251)
0927 02:42:59.087698 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:59.087714 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:59.087749 (+ 35us) server_negotiation.cc:407] Connection header received
0927 02:42:59.088399 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:59.088416 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:59.088774 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:59.089096 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:59.089948 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:59.091107 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:59.092296 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:59.092807 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:59.092931 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:59.095853 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:59.095859 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:59.095878 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:59.095912 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:59.097594 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:59.098215 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:59.098221 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:59.098226 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:59.098386 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:59.098860 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:59.098863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:59.098865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:59.099394 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:59.100344 (+ 950us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:59.100538 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 02:42:59.100791 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":128,"threads_started":1}
W20260927 02:42:59.104916 21752 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 37356d01fa5d4cdf9fad29a6245133b2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:42:59.104990 21752 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:42:59.105080 21746 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:42:59.105338 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:42:59.105886 21746 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:42:59.112532 21746 catalog_manager.cc:1347] Generated new cluster ID: 621b838ed9f24e2a818f4bebae951735
I20260927 02:42:59.112597 21746 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:42:59.115619 21735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:42:59.107982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:47009 (local address 127.0.0.1:33251)
0927 02:42:59.108086 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 02:42:59.108090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:42:59.108104 (+ 14us) server_negotiation.cc:407] Connection header received
0927 02:42:59.108149 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:42:59.108152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:42:59.108198 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:42:59.108281 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:42:59.108989 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:59.109501 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:59.110282 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:42:59.110525 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:42:59.110611 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:42:59.112224 (+ 1613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:42:59.112228 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:42:59.112231 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:42:59.112255 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:42:59.113855 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:59.114358 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:59.114361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:59.114363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:59.114460 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:42:59.114799 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:42:59.114802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:42:59.114804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:42:59.115011 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:42:59.115160 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 02:42:59.115371 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 02:42:59.115466 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 02:42:59.121939 21746 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:42:59.122838 21746 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:42:59.131525 21746 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37356d01fa5d4cdf9fad29a6245133b2: Generated new TSK 0
I20260927 02:42:59.132223 21746 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:42:59.138501 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 21664
W20260927 02:42:59.144876 21649 connection.cc:535] client connection to 127.0.0.1:33251 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33251): Connection reset by peer (error 104)
2026-09-27T02:42:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1285 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476718023131-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:42:59 dist-test-slave-zv8z krb5kdc[21763](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:42:59 dist-test-slave-zv8z krb5kdc[21763](info): set up 2 sockets
Sep 27 02:42:59 dist-test-slave-zv8z krb5kdc[21763](info): commencing operation
krb5kdc: starting...
W20260927 02:43:00.090196 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476718023131-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476718023131-392-0/krb5kdc/test-user.keytab.
Sep 27 02:43:00 dist-test-slave-zv8z krb5kdc[21763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:43:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:43:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476718023131-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476718023131-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:43:00.201078 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45103
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476718023131-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45103
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:43:00.306387 21776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:43:00.306650 21776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:43:00.306722 21776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:43:00.310770 21776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:43:00.310827 21776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:43:00.310853 21776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:43:00.310882 21776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:43:00.310910 21776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:43:00.316120 21776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45103
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476718023131-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:43:00.317095 21776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:43:00.318058 21776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:43:00.325919 21781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:00.325929 21784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:00.325929 21782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:43:00.325980 21776 server_base.cc:1034] running on GCE node
I20260927 02:43:00.326603 21776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:43:00.327572 21776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:43:00.328713 21776 hybrid_clock.cc:648] HybridClock initialized: now 1790476980328688 us; error 37 us; skew 500 ppm
Sep 27 02:43:00 dist-test-slave-zv8z krb5kdc[21763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:43:00.336014 21776 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:43:00.336858 21776 webserver.cc:460] Webserver started at http://127.0.0.1:45317/ using document root <none> and password file <none>
I20260927 02:43:00.337363 21776 fs_manager.cc:359] Metadata directory not provided
I20260927 02:43:00.337419 21776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:43:00.337584 21776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:43:00.339607 21776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476718023131-392-0/minicluster-data/master-0/data/instance:
uuid: "cec76401f4ed4478978ae38562d52080"
format_stamp: "Formatted at 2026-09-27 02:43:00 on dist-test-slave-zv8z"
server_key: "f802bb4fcbeebe6b933b3a131dc086fd"
server_key_iv: "b6cb86965f9ae3429a65f356c5cebef0"
server_key_version: "encryptionkey@0"
I20260927 02:43:00.340088 21776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476718023131-392-0/minicluster-data/master-0/wal/instance:
uuid: "cec76401f4ed4478978ae38562d52080"
format_stamp: "Formatted at 2026-09-27 02:43:00 on dist-test-slave-zv8z"
server_key: "f802bb4fcbeebe6b933b3a131dc086fd"
server_key_iv: "b6cb86965f9ae3429a65f356c5cebef0"
server_key_version: "encryptionkey@0"
I20260927 02:43:00.343505 21776 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260927 02:43:00.345724 21791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:00.346946 21776 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:43:00.347045 21776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "cec76401f4ed4478978ae38562d52080"
format_stamp: "Formatted at 2026-09-27 02:43:00 on dist-test-slave-zv8z"
server_key: "f802bb4fcbeebe6b933b3a131dc086fd"
server_key_iv: "b6cb86965f9ae3429a65f356c5cebef0"
server_key_version: "encryptionkey@0"
I20260927 02:43:00.347159 21776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:43:00.370430 21776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:43:00.373513 21776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:43:00.373682 21776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:43:00.380419 21776 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45103
I20260927 02:43:00.380457 21843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45103 every 8 connection(s)
I20260927 02:43:00.381331 21776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:43:00.383998 21844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:43:00.386699 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 21776
I20260927 02:43:00.386837 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790476718023131-392-0/minicluster-data/master-0/wal/instance
I20260927 02:43:00.387120 392 external_mini_cluster.cc:1428] Setting key d2289165e1c49441b911103937eaacd7
I20260927 02:43:00.389859 21844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cec76401f4ed4478978ae38562d52080: Bootstrap starting.
I20260927 02:43:00.392409 21844 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cec76401f4ed4478978ae38562d52080: Neither blocks nor log segments found. Creating new log.
I20260927 02:43:00.393105 21844 log.cc:824] T 00000000000000000000000000000000 P cec76401f4ed4478978ae38562d52080: Log is configured to *not* fsync() on all Append() calls
I20260927 02:43:00.395080 21844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cec76401f4ed4478978ae38562d52080: No bootstrap required, opened a new log
I20260927 02:43:00.396456 21847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:00.388463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:56551 (local address 127.0.0.1:45103)
0927 02:43:00.388863 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:00.388881 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:00.388923 (+ 42us) server_negotiation.cc:407] Connection header received
0927 02:43:00.389707 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:00.389729 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:00.390146 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:00.390508 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:00.390908 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:00.392327 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:00.393146 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:00.393722 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:00.393871 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:00.394627 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:00.394633 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:00.394647 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:43:00.395098 (+ 451us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:00.395165 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:43:00.395194 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:00.395315 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:00.395471 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 02:43:00.395605 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":103,"threads_started":1}
I20260927 02:43:00.398728 21844 raft_consensus.cc:348] T 00000000000000000000000000000000 P cec76401f4ed4478978ae38562d52080 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec76401f4ed4478978ae38562d52080" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45103 } }
I20260927 02:43:00.398929 21844 raft_consensus.cc:374] T 00000000000000000000000000000000 P cec76401f4ed4478978ae38562d52080 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:43:00.398984 21844 raft_consensus.cc:729] T 00000000000000000000000000000000 P cec76401f4ed4478978ae38562d52080 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec76401f4ed4478978ae38562d52080, State: Initialized, Role: FOLLOWER
I20260927 02:43:00.399081 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:43:00.399377 21844 consensus_queue.cc:260] T 00000000000000000000000000000000 P cec76401f4ed4478978ae38562d52080 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec76401f4ed4478978ae38562d52080" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45103 } }
I20260927 02:43:00.399493 21844 raft_consensus.cc:388] T 00000000000000000000000000000000 P cec76401f4ed4478978ae38562d52080 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:43:00.399550 21844 raft_consensus.cc:482] T 00000000000000000000000000000000 P cec76401f4ed4478978ae38562d52080 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:43:00.399636 21844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cec76401f4ed4478978ae38562d52080 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:43:00.400451 21844 raft_consensus.cc:504] T 00000000000000000000000000000000 P cec76401f4ed4478978ae38562d52080 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec76401f4ed4478978ae38562d52080" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45103 } }
I20260927 02:43:00.400743 21844 leader_election.cc:302] T 00000000000000000000000000000000 P cec76401f4ed4478978ae38562d52080 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cec76401f4ed4478978ae38562d52080; no voters:
I20260927 02:43:00.401026 21844 leader_election.cc:288] T 00000000000000000000000000000000 P cec76401f4ed4478978ae38562d52080 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:43:00.401090 21854 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cec76401f4ed4478978ae38562d52080 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:43:00.401279 21854 raft_consensus.cc:686] T 00000000000000000000000000000000 P cec76401f4ed4478978ae38562d52080 [term 1 LEADER]: Becoming Leader. State: Replica: cec76401f4ed4478978ae38562d52080, State: Running, Role: LEADER
I20260927 02:43:00.401584 21854 consensus_queue.cc:237] T 00000000000000000000000000000000 P cec76401f4ed4478978ae38562d52080 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec76401f4ed4478978ae38562d52080" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45103 } }
I20260927 02:43:00.402264 21844 sys_catalog.cc:564] T 00000000000000000000000000000000 P cec76401f4ed4478978ae38562d52080 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:43:00.402653 21854 sys_catalog.cc:455] T 00000000000000000000000000000000 P cec76401f4ed4478978ae38562d52080 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cec76401f4ed4478978ae38562d52080. Latest consensus state: current_term: 1 leader_uuid: "cec76401f4ed4478978ae38562d52080" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec76401f4ed4478978ae38562d52080" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45103 } } }
I20260927 02:43:00.402762 21854 sys_catalog.cc:458] T 00000000000000000000000000000000 P cec76401f4ed4478978ae38562d52080 [sys.catalog]: This master's current role is: LEADER
I20260927 02:43:00.404484 21856 sys_catalog.cc:455] T 00000000000000000000000000000000 P cec76401f4ed4478978ae38562d52080 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cec76401f4ed4478978ae38562d52080" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec76401f4ed4478978ae38562d52080" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45103 } } }
I20260927 02:43:00.404603 21856 sys_catalog.cc:458] T 00000000000000000000000000000000 P cec76401f4ed4478978ae38562d52080 [sys.catalog]: This master's current role is: LEADER
I20260927 02:43:00.404713 21858 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:43:00.405495 21847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:00.400931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:58673 (local address 127.0.0.1:45103)
0927 02:43:00.401038 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:00.401041 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:00.401281 (+ 240us) server_negotiation.cc:407] Connection header received
0927 02:43:00.401423 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:00.401426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:00.401472 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:00.401543 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:00.401907 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:00.402653 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:00.403783 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:00.404091 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:00.404184 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:00.405120 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:00.405124 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:00.405127 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:43:00.405139 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:00.405163 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:43:00.405175 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:00.405261 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:00.405348 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 02:43:00.405379 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 02:43:00.407536 21858 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:43:00.412949 21858 catalog_manager.cc:1347] Generated new cluster ID: 8abdb3ffd4344692be87247acfbc89fe
I20260927 02:43:00.412997 21858 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:43:00.420462 21858 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:43:00.421273 21858 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:43:00.427608 21858 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cec76401f4ed4478978ae38562d52080: Generated new TSK 0
I20260927 02:43:00.428354 21858 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:43:00.437516 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 21776
W20260927 02:43:00.443786 21761 connection.cc:535] client connection to 127.0.0.1:45103 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45103): Connection reset by peer (error 104)
2026-09-27T02:43:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1299 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476718023131-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:43:00 dist-test-slave-zv8z krb5kdc[21875](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:43:00 dist-test-slave-zv8z krb5kdc[21875](info): set up 2 sockets
Sep 27 02:43:00 dist-test-slave-zv8z krb5kdc[21875](info): commencing operation
krb5kdc: starting...
W20260927 02:43:01.382159 392 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476718023131-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476718023131-392-0/krb5kdc/test-user.keytab.
Sep 27 02:43:01 dist-test-slave-zv8z krb5kdc[21875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:43:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:43:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476718023131-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476718023131-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:43:01.497713 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38593
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476718023131-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:43:01.603929 21888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:43:01.604203 21888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:43:01.604261 21888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:43:01.608186 21888 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:43:01.608246 21888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:43:01.608273 21888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:43:01.608295 21888 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:43:01.608321 21888 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:43:01.613508 21888 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38593
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476718023131-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:43:01.614569 21888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:43:01.615558 21888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:43:01.623831 21893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:01.623926 21896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:43:01.624034 21888 server_base.cc:1034] running on GCE node
W20260927 02:43:01.623926 21894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:43:01.624629 21888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:43:01.625701 21888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:43:01.626864 21888 hybrid_clock.cc:648] HybridClock initialized: now 1790476981626849 us; error 27 us; skew 500 ppm
Sep 27 02:43:01 dist-test-slave-zv8z krb5kdc[21875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:43:01.629747 21888 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:43:01.630636 21888 webserver.cc:460] Webserver started at http://127.0.0.1:42707/ using document root <none> and password file <none>
I20260927 02:43:01.631165 21888 fs_manager.cc:359] Metadata directory not provided
I20260927 02:43:01.631224 21888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:43:01.631409 21888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:43:01.633430 21888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476718023131-392-0/minicluster-data/master-0/data/instance:
uuid: "fc8e3ad270794db991ec88e828dbe7d9"
format_stamp: "Formatted at 2026-09-27 02:43:01 on dist-test-slave-zv8z"
server_key: "47f99674d42e69bb27d49e90f725b3a2"
server_key_iv: "4a3595e6924d585c5110e42f625c546e"
server_key_version: "encryptionkey@0"
I20260927 02:43:01.633955 21888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476718023131-392-0/minicluster-data/master-0/wal/instance:
uuid: "fc8e3ad270794db991ec88e828dbe7d9"
format_stamp: "Formatted at 2026-09-27 02:43:01 on dist-test-slave-zv8z"
server_key: "47f99674d42e69bb27d49e90f725b3a2"
server_key_iv: "4a3595e6924d585c5110e42f625c546e"
server_key_version: "encryptionkey@0"
I20260927 02:43:01.637581 21888 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 02:43:01.640008 21903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:01.641237 21888 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:43:01.641350 21888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "fc8e3ad270794db991ec88e828dbe7d9"
format_stamp: "Formatted at 2026-09-27 02:43:01 on dist-test-slave-zv8z"
server_key: "47f99674d42e69bb27d49e90f725b3a2"
server_key_iv: "4a3595e6924d585c5110e42f625c546e"
server_key_version: "encryptionkey@0"
I20260927 02:43:01.641445 21888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:43:01.645957 21888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:43:01.648743 21888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:43:01.648900 21888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:43:01.655732 21888 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38593
I20260927 02:43:01.655762 21955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38593 every 8 connection(s)
I20260927 02:43:01.656705 21888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:43:01.659973 21956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:43:01.663203 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 21888
I20260927 02:43:01.663354 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790476718023131-392-0/minicluster-data/master-0/wal/instance
I20260927 02:43:01.663605 392 external_mini_cluster.cc:1428] Setting key 6dd3bc5efe0443910dfeb4badd0f9988
I20260927 02:43:01.667241 21956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc8e3ad270794db991ec88e828dbe7d9: Bootstrap starting.
I20260927 02:43:01.669663 21959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:01.665059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:41969 (local address 127.0.0.1:38593)
0927 02:43:01.665514 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:01.665532 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:01.665574 (+ 42us) server_negotiation.cc:407] Connection header received
0927 02:43:01.666391 (+ 817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:01.666415 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:01.666773 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:01.667133 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:01.668238 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:01.668243 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:01.668257 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:43:01.668270 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:01.668340 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:43:01.668369 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:01.668509 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:01.668624 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 02:43:01.668779 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":147,"threads_started":1}
I20260927 02:43:01.669906 21956 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc8e3ad270794db991ec88e828dbe7d9: Neither blocks nor log segments found. Creating new log.
I20260927 02:43:01.670778 21956 log.cc:824] T 00000000000000000000000000000000 P fc8e3ad270794db991ec88e828dbe7d9: Log is configured to *not* fsync() on all Append() calls
I20260927 02:43:01.672605 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:43:01.672835 21956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc8e3ad270794db991ec88e828dbe7d9: No bootstrap required, opened a new log
I20260927 02:43:01.676105 21959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:01.674581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.113:42617 (local address 127.0.0.1:38593)
0927 02:43:01.674682 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:01.674687 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:01.674858 (+ 171us) server_negotiation.cc:407] Connection header received
0927 02:43:01.674971 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:01.674975 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:01.675006 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:01.675072 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:01.675741 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:01.675745 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:01.675748 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:43:01.675759 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:01.675786 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:43:01.675796 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:01.675864 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:01.675949 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 02:43:01.675986 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 02:43:01.676750 21956 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc8e3ad270794db991ec88e828dbe7d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc8e3ad270794db991ec88e828dbe7d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38593 } }
I20260927 02:43:01.677019 21956 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc8e3ad270794db991ec88e828dbe7d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:43:01.677079 21956 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc8e3ad270794db991ec88e828dbe7d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc8e3ad270794db991ec88e828dbe7d9, State: Initialized, Role: FOLLOWER
I20260927 02:43:01.677508 21956 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc8e3ad270794db991ec88e828dbe7d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc8e3ad270794db991ec88e828dbe7d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38593 } }
I20260927 02:43:01.677628 21956 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc8e3ad270794db991ec88e828dbe7d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:43:01.677670 21956 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc8e3ad270794db991ec88e828dbe7d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:43:01.677760 21956 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc8e3ad270794db991ec88e828dbe7d9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:43:01.678725 21956 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc8e3ad270794db991ec88e828dbe7d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc8e3ad270794db991ec88e828dbe7d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38593 } }
I20260927 02:43:01.679032 21956 leader_election.cc:302] T 00000000000000000000000000000000 P fc8e3ad270794db991ec88e828dbe7d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc8e3ad270794db991ec88e828dbe7d9; no voters:
I20260927 02:43:01.679301 21956 leader_election.cc:288] T 00000000000000000000000000000000 P fc8e3ad270794db991ec88e828dbe7d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:43:01.679407 21966 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc8e3ad270794db991ec88e828dbe7d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:43:01.679865 21966 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc8e3ad270794db991ec88e828dbe7d9 [term 1 LEADER]: Becoming Leader. State: Replica: fc8e3ad270794db991ec88e828dbe7d9, State: Running, Role: LEADER
I20260927 02:43:01.680222 21966 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc8e3ad270794db991ec88e828dbe7d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc8e3ad270794db991ec88e828dbe7d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38593 } }
I20260927 02:43:01.680495 21956 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc8e3ad270794db991ec88e828dbe7d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:43:01.681730 21968 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc8e3ad270794db991ec88e828dbe7d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc8e3ad270794db991ec88e828dbe7d9. Latest consensus state: current_term: 1 leader_uuid: "fc8e3ad270794db991ec88e828dbe7d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc8e3ad270794db991ec88e828dbe7d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38593 } } }
I20260927 02:43:01.681856 21968 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc8e3ad270794db991ec88e828dbe7d9 [sys.catalog]: This master's current role is: LEADER
I20260927 02:43:01.682165 21967 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc8e3ad270794db991ec88e828dbe7d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc8e3ad270794db991ec88e828dbe7d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc8e3ad270794db991ec88e828dbe7d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38593 } } }
I20260927 02:43:01.682273 21967 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc8e3ad270794db991ec88e828dbe7d9 [sys.catalog]: This master's current role is: LEADER
I20260927 02:43:01.682384 21977 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:43:01.684993 21977 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:43:01.690109 21977 catalog_manager.cc:1347] Generated new cluster ID: 22ee9bfb6630497f88f25420f9481601
I20260927 02:43:01.690166 21977 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:43:01.728679 21977 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:43:01.729053 21977 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:43:01.734583 21977 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc8e3ad270794db991ec88e828dbe7d9: Generated new TSK 0
I20260927 02:43:01.735262 21977 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:43:01.745370 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 21888
2026-09-27T02:43:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1305 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476718023131-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:43:01 dist-test-slave-zv8z krb5kdc[21987](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:43:01 dist-test-slave-zv8z krb5kdc[21987](info): set up 2 sockets
Sep 27 02:43:01 dist-test-slave-zv8z krb5kdc[21987](info): commencing operation
krb5kdc: starting...
W20260927 02:43:02.694231 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476718023131-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476718023131-392-0/krb5kdc/test-user.keytab.
Sep 27 02:43:02 dist-test-slave-zv8z krb5kdc[21987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:43:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:43:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:43:02.804623 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41599
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:43:02.908128 22000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:43:02.908375 22000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:43:02.908414 22000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:43:02.912348 22000 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:43:02.912400 22000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:43:02.912417 22000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:43:02.912432 22000 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:43:02.912446 22000 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:43:02.917543 22000 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41599
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:43:02.918503 22000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:43:02.919431 22000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:43:02.927397 22005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:02.927397 22006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:02.927403 22008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:43:02.927644 22000 server_base.cc:1034] running on GCE node
I20260927 02:43:02.928131 22000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:43:02.929136 22000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:43:02.930305 22000 hybrid_clock.cc:648] HybridClock initialized: now 1790476982930253 us; error 68 us; skew 500 ppm
Sep 27 02:43:02 dist-test-slave-zv8z krb5kdc[21987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476982, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:43:02.936447 22000 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:43:02.937276 22000 webserver.cc:460] Webserver started at http://127.0.98.62:45495/ using document root <none> and password file <none>
I20260927 02:43:02.937777 22000 fs_manager.cc:359] Metadata directory not provided
I20260927 02:43:02.937839 22000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:43:02.938035 22000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:43:02.940037 22000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476718023131-392-0/minicluster-data/master-0/data/instance:
uuid: "506070455ec941c5bd715e596337a96b"
format_stamp: "Formatted at 2026-09-27 02:43:02 on dist-test-slave-zv8z"
server_key: "505c3d205c6eb80a0a783bab2742c5c0"
server_key_iv: "20b1deb351636afe7fc9f9d24387886e"
server_key_version: "encryptionkey@0"
I20260927 02:43:02.940543 22000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476718023131-392-0/minicluster-data/master-0/wal/instance:
uuid: "506070455ec941c5bd715e596337a96b"
format_stamp: "Formatted at 2026-09-27 02:43:02 on dist-test-slave-zv8z"
server_key: "505c3d205c6eb80a0a783bab2742c5c0"
server_key_iv: "20b1deb351636afe7fc9f9d24387886e"
server_key_version: "encryptionkey@0"
I20260927 02:43:02.944000 22000 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:43:02.946313 22015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:02.947431 22000 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:43:02.947540 22000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "506070455ec941c5bd715e596337a96b"
format_stamp: "Formatted at 2026-09-27 02:43:02 on dist-test-slave-zv8z"
server_key: "505c3d205c6eb80a0a783bab2742c5c0"
server_key_iv: "20b1deb351636afe7fc9f9d24387886e"
server_key_version: "encryptionkey@0"
I20260927 02:43:02.947638 22000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:43:02.966718 22000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:43:02.969633 22000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:43:02.969801 22000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:43:02.976290 22000 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41599
I20260927 02:43:02.976305 22067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41599 every 8 connection(s)
I20260927 02:43:02.977180 22000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:43:02.979794 22068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:43:02.979928 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 22000
I20260927 02:43:02.980067 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790476718023131-392-0/minicluster-data/master-0/wal/instance
I20260927 02:43:02.980348 392 external_mini_cluster.cc:1428] Setting key 7a76170a76449220205211810d68efea
I20260927 02:43:02.986021 22068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 506070455ec941c5bd715e596337a96b: Bootstrap starting.
I20260927 02:43:02.988502 22068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 506070455ec941c5bd715e596337a96b: Neither blocks nor log segments found. Creating new log.
Sep 27 02:43:02 dist-test-slave-zv8z krb5kdc[21987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:43:02.989197 22068 log.cc:824] T 00000000000000000000000000000000 P 506070455ec941c5bd715e596337a96b: Log is configured to *not* fsync() on all Append() calls
I20260927 02:43:02.991081 22068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 506070455ec941c5bd715e596337a96b: No bootstrap required, opened a new log
I20260927 02:43:02.993677 22071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:02.981660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43349 (local address 127.0.98.62:41599)
0927 02:43:02.982061 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:02.982075 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:02.982108 (+ 33us) server_negotiation.cc:407] Connection header received
0927 02:43:02.982800 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:02.982817 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:02.983149 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:02.983410 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:02.984236 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:02.985191 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:02.986335 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:02.986792 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:02.986912 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:02.989565 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:02.989570 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:02.989583 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:02.989609 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:02.991121 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:02.991573 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:02.991578 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:02.991582 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:02.991684 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:02.991978 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:02.991981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:02.991982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:02.992333 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:02.992528 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:02.992822 (+ 294us) server_negotiation.cc:299] Negotiation successful
0927 02:43:02.993010 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":123,"threads_started":1}
I20260927 02:43:02.994609 22068 raft_consensus.cc:348] T 00000000000000000000000000000000 P 506070455ec941c5bd715e596337a96b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "506070455ec941c5bd715e596337a96b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41599 } }
I20260927 02:43:02.994829 22068 raft_consensus.cc:374] T 00000000000000000000000000000000 P 506070455ec941c5bd715e596337a96b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:43:02.994891 22068 raft_consensus.cc:729] T 00000000000000000000000000000000 P 506070455ec941c5bd715e596337a96b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 506070455ec941c5bd715e596337a96b, State: Initialized, Role: FOLLOWER
I20260927 02:43:02.995303 22068 consensus_queue.cc:260] T 00000000000000000000000000000000 P 506070455ec941c5bd715e596337a96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "506070455ec941c5bd715e596337a96b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41599 } }
I20260927 02:43:02.995445 22068 raft_consensus.cc:388] T 00000000000000000000000000000000 P 506070455ec941c5bd715e596337a96b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:43:02.995512 22068 raft_consensus.cc:482] T 00000000000000000000000000000000 P 506070455ec941c5bd715e596337a96b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:43:02.995602 22068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 506070455ec941c5bd715e596337a96b [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:43:02.996480 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:43:02.996585 22068 raft_consensus.cc:504] T 00000000000000000000000000000000 P 506070455ec941c5bd715e596337a96b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "506070455ec941c5bd715e596337a96b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41599 } }
I20260927 02:43:02.996923 22068 leader_election.cc:302] T 00000000000000000000000000000000 P 506070455ec941c5bd715e596337a96b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 506070455ec941c5bd715e596337a96b; no voters:
I20260927 02:43:02.997226 22068 leader_election.cc:288] T 00000000000000000000000000000000 P 506070455ec941c5bd715e596337a96b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:43:02.997310 22073 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 506070455ec941c5bd715e596337a96b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:43:02.997502 22073 raft_consensus.cc:686] T 00000000000000000000000000000000 P 506070455ec941c5bd715e596337a96b [term 1 LEADER]: Becoming Leader. State: Replica: 506070455ec941c5bd715e596337a96b, State: Running, Role: LEADER
I20260927 02:43:02.997848 22073 consensus_queue.cc:237] T 00000000000000000000000000000000 P 506070455ec941c5bd715e596337a96b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "506070455ec941c5bd715e596337a96b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41599 } }
I20260927 02:43:02.998540 22068 sys_catalog.cc:564] T 00000000000000000000000000000000 P 506070455ec941c5bd715e596337a96b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:43:02.999475 22075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 506070455ec941c5bd715e596337a96b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 506070455ec941c5bd715e596337a96b. Latest consensus state: current_term: 1 leader_uuid: "506070455ec941c5bd715e596337a96b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "506070455ec941c5bd715e596337a96b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41599 } } }
I20260927 02:43:02.999590 22075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 506070455ec941c5bd715e596337a96b [sys.catalog]: This master's current role is: LEADER
I20260927 02:43:02.999895 22074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 506070455ec941c5bd715e596337a96b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "506070455ec941c5bd715e596337a96b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "506070455ec941c5bd715e596337a96b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41599 } } }
I20260927 02:43:02.999948 22087 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:43:02.999996 22074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 506070455ec941c5bd715e596337a96b [sys.catalog]: This master's current role is: LEADER
I20260927 02:43:03.003345 22087 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:43:03.008908 22071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:02.998653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50321 (local address 127.0.98.62:41599)
0927 02:43:02.998746 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:02.998750 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:03.000558 (+ 1808us) server_negotiation.cc:407] Connection header received
0927 02:43:03.000711 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:03.000715 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:03.000758 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:03.000825 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:03.001594 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:03.002142 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:03.003257 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:03.003605 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:03.003700 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:03.005162 (+ 1462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:03.005166 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:03.005169 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:03.005195 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:03.006867 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:03.007490 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:03.007494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:03.007496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:03.007584 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:03.007942 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:03.007946 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:03.007948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:03.008126 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:03.008262 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:03.008516 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 02:43:03.008602 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 02:43:03.009068 22087 catalog_manager.cc:1347] Generated new cluster ID: e23159f4361547d5b324c92b096453d3
I20260927 02:43:03.009123 22087 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:43:03.019833 22087 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:43:03.020985 22087 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:43:03.028656 22087 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 506070455ec941c5bd715e596337a96b: Generated new TSK 0
I20260927 02:43:03.029408 22087 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:43:03.033334 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 22000
W20260927 02:43:03.037974 21985 connection.cc:535] client connection to 127.0.98.62:41599 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:41599): Connection reset by peer (error 104)
2026-09-27T02:43:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1288 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476718023131-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:43:03 dist-test-slave-zv8z krb5kdc[22099](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:43:03 dist-test-slave-zv8z krb5kdc[22099](info): set up 2 sockets
Sep 27 02:43:03 dist-test-slave-zv8z krb5kdc[22099](info): commencing operation
krb5kdc: starting...
W20260927 02:43:03.986141 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476718023131-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476718023131-392-0/krb5kdc/test-user.keytab.
Sep 27 02:43:04 dist-test-slave-zv8z krb5kdc[22099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:43:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:43:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:43:04.097532 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37287
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:43:04.204030 22112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:43:04.204283 22112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:43:04.204339 22112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:43:04.208261 22112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:43:04.208314 22112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:43:04.208340 22112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:43:04.208359 22112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:43:04.208380 22112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:43:04.213470 22112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37287
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:43:04.214536 22112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:43:04.215545 22112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:43:04.223855 22118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:04.223860 22117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:04.223860 22120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:43:04.224187 22112 server_base.cc:1034] running on GCE node
I20260927 02:43:04.224730 22112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:43:04.225775 22112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:43:04.226930 22112 hybrid_clock.cc:648] HybridClock initialized: now 1790476984226914 us; error 33 us; skew 500 ppm
Sep 27 02:43:04 dist-test-slave-zv8z krb5kdc[22099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476984, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:43:04.229475 22112 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:43:04.230345 22112 webserver.cc:460] Webserver started at http://127.0.98.62:46417/ using document root <none> and password file <none>
I20260927 02:43:04.230873 22112 fs_manager.cc:359] Metadata directory not provided
I20260927 02:43:04.230937 22112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:43:04.231110 22112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:43:04.233086 22112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476718023131-392-0/minicluster-data/master-0/data/instance:
uuid: "2a46c1e6edd64b8094d2032c7d330f61"
format_stamp: "Formatted at 2026-09-27 02:43:04 on dist-test-slave-zv8z"
server_key: "8ade7df0350ceaa806191fcfa7a5fed1"
server_key_iv: "76c4fe426542cc60b83797f77e2ccf63"
server_key_version: "encryptionkey@0"
I20260927 02:43:04.233633 22112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476718023131-392-0/minicluster-data/master-0/wal/instance:
uuid: "2a46c1e6edd64b8094d2032c7d330f61"
format_stamp: "Formatted at 2026-09-27 02:43:04 on dist-test-slave-zv8z"
server_key: "8ade7df0350ceaa806191fcfa7a5fed1"
server_key_iv: "76c4fe426542cc60b83797f77e2ccf63"
server_key_version: "encryptionkey@0"
I20260927 02:43:04.237139 22112 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 02:43:04.239549 22127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:04.240803 22112 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:43:04.240914 22112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "2a46c1e6edd64b8094d2032c7d330f61"
format_stamp: "Formatted at 2026-09-27 02:43:04 on dist-test-slave-zv8z"
server_key: "8ade7df0350ceaa806191fcfa7a5fed1"
server_key_iv: "76c4fe426542cc60b83797f77e2ccf63"
server_key_version: "encryptionkey@0"
I20260927 02:43:04.241010 22112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:43:04.252347 22112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:43:04.255038 22112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:43:04.255204 22112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:43:04.261652 22112 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37287
I20260927 02:43:04.261682 22179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37287 every 8 connection(s)
I20260927 02:43:04.262580 22112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:43:04.263280 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 22112
I20260927 02:43:04.263402 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790476718023131-392-0/minicluster-data/master-0/wal/instance
I20260927 02:43:04.263653 392 external_mini_cluster.cc:1428] Setting key a0f457da1f26c0822c3335e58d8fd4fb
I20260927 02:43:04.265683 22180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:43:04.272646 22180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a46c1e6edd64b8094d2032c7d330f61: Bootstrap starting.
I20260927 02:43:04.273780 22183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:04.264981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52905 (local address 127.0.98.62:37287)
0927 02:43:04.265397 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:04.265415 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:04.265457 (+ 42us) server_negotiation.cc:407] Connection header received
0927 02:43:04.266304 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:04.266327 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:04.266748 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:04.267112 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:04.267507 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:04.269013 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:04.270322 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:04.270991 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:04.271158 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:04.271981 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:04.271988 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:04.272003 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:43:04.272494 (+ 491us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:04.272559 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:43:04.272589 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:04.272722 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:04.272940 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 02:43:04.273066 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":123,"threads_started":1}
I20260927 02:43:04.274948 22180 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a46c1e6edd64b8094d2032c7d330f61: Neither blocks nor log segments found. Creating new log.
I20260927 02:43:04.275676 22180 log.cc:824] T 00000000000000000000000000000000 P 2a46c1e6edd64b8094d2032c7d330f61: Log is configured to *not* fsync() on all Append() calls
I20260927 02:43:04.276491 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:43:04.277518 22180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a46c1e6edd64b8094d2032c7d330f61: No bootstrap required, opened a new log
I20260927 02:43:04.280895 22180 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a46c1e6edd64b8094d2032c7d330f61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a46c1e6edd64b8094d2032c7d330f61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37287 } }
I20260927 02:43:04.281122 22180 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a46c1e6edd64b8094d2032c7d330f61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:43:04.281190 22180 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a46c1e6edd64b8094d2032c7d330f61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a46c1e6edd64b8094d2032c7d330f61, State: Initialized, Role: FOLLOWER
I20260927 02:43:04.281771 22180 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a46c1e6edd64b8094d2032c7d330f61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a46c1e6edd64b8094d2032c7d330f61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37287 } }
I20260927 02:43:04.281934 22180 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a46c1e6edd64b8094d2032c7d330f61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:43:04.281997 22180 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a46c1e6edd64b8094d2032c7d330f61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:43:04.282102 22180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a46c1e6edd64b8094d2032c7d330f61 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:43:04.283089 22180 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a46c1e6edd64b8094d2032c7d330f61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a46c1e6edd64b8094d2032c7d330f61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37287 } }
I20260927 02:43:04.283439 22180 leader_election.cc:302] T 00000000000000000000000000000000 P 2a46c1e6edd64b8094d2032c7d330f61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a46c1e6edd64b8094d2032c7d330f61; no voters:
I20260927 02:43:04.283759 22180 leader_election.cc:288] T 00000000000000000000000000000000 P 2a46c1e6edd64b8094d2032c7d330f61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:43:04.283839 22190 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a46c1e6edd64b8094d2032c7d330f61 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:43:04.284042 22190 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a46c1e6edd64b8094d2032c7d330f61 [term 1 LEADER]: Becoming Leader. State: Replica: 2a46c1e6edd64b8094d2032c7d330f61, State: Running, Role: LEADER
I20260927 02:43:04.284401 22190 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a46c1e6edd64b8094d2032c7d330f61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a46c1e6edd64b8094d2032c7d330f61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37287 } }
I20260927 02:43:04.284938 22180 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a46c1e6edd64b8094d2032c7d330f61 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:43:04.285547 22183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:04.278290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55137 (local address 127.0.98.62:37287)
0927 02:43:04.278390 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:04.278394 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:04.281395 (+ 3001us) server_negotiation.cc:407] Connection header received
0927 02:43:04.281577 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:04.281579 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:04.281618 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:04.281708 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:04.282057 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:04.282666 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:04.283743 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:04.284052 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:04.284566 (+ 514us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:04.285096 (+ 530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:04.285099 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:04.285102 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:43:04.285114 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:04.285140 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:43:04.285153 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:04.285229 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:04.285384 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 02:43:04.285420 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 02:43:04.286281 22191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a46c1e6edd64b8094d2032c7d330f61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a46c1e6edd64b8094d2032c7d330f61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a46c1e6edd64b8094d2032c7d330f61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37287 } } }
I20260927 02:43:04.286396 22191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a46c1e6edd64b8094d2032c7d330f61 [sys.catalog]: This master's current role is: LEADER
I20260927 02:43:04.286762 22192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a46c1e6edd64b8094d2032c7d330f61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a46c1e6edd64b8094d2032c7d330f61. Latest consensus state: current_term: 1 leader_uuid: "2a46c1e6edd64b8094d2032c7d330f61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a46c1e6edd64b8094d2032c7d330f61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37287 } } }
I20260927 02:43:04.286931 22192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a46c1e6edd64b8094d2032c7d330f61 [sys.catalog]: This master's current role is: LEADER
W20260927 02:43:04.289512 22205 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2a46c1e6edd64b8094d2032c7d330f61: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:43:04.289592 22205 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:43:04.289657 22202 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:43:04.290393 22202 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:43:04.295251 22202 catalog_manager.cc:1347] Generated new cluster ID: 699ac2b09324441aad31a2c62f7bf302
I20260927 02:43:04.295318 22202 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:43:04.317693 22202 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:43:04.318997 22202 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:43:04.325016 22202 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a46c1e6edd64b8094d2032c7d330f61: Generated new TSK 0
I20260927 02:43:04.325835 22202 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:43:04.329365 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 22112
W20260927 02:43:04.334125 22097 connection.cc:535] client connection to 127.0.98.62:37287 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:37287): Connection reset by peer (error 104)
2026-09-27T02:43:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1296 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476718023131-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:43:04 dist-test-slave-zv8z krb5kdc[22211](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:43:04 dist-test-slave-zv8z krb5kdc[22211](info): set up 2 sockets
Sep 27 02:43:04 dist-test-slave-zv8z krb5kdc[22211](info): commencing operation
krb5kdc: starting...
W20260927 02:43:05.262378 392 mini_kdc.cc:121] Time spent starting KDC: real 0.914s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476718023131-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476718023131-392-0/krb5kdc/test-user.keytab.
Sep 27 02:43:05 dist-test-slave-zv8z krb5kdc[22211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:43:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:43:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:43:05.372918 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40397
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:43:05.478849 22224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:43:05.479112 22224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:43:05.479171 22224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:43:05.483152 22224 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:43:05.483206 22224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:43:05.483232 22224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:43:05.483254 22224 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:43:05.483282 22224 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:43:05.488395 22224 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40397
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:43:05.489373 22224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:43:05.490375 22224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:43:05.498445 22224 server_base.cc:1034] running on GCE node
W20260927 02:43:05.498402 22232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:05.498415 22229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:05.498404 22230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:43:05.499219 22224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:43:05.500247 22224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:43:05.501407 22224 hybrid_clock.cc:648] HybridClock initialized: now 1790476985501349 us; error 71 us; skew 500 ppm
Sep 27 02:43:05 dist-test-slave-zv8z krb5kdc[22211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:43:05.503928 22224 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:43:05.504709 22224 webserver.cc:460] Webserver started at http://127.0.98.62:43717/ using document root <none> and password file <none>
I20260927 02:43:05.505185 22224 fs_manager.cc:359] Metadata directory not provided
I20260927 02:43:05.505237 22224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:43:05.505400 22224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:43:05.507318 22224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476718023131-392-0/minicluster-data/master-0/data/instance:
uuid: "9b9426a255b34f94bd4e61a45078315d"
format_stamp: "Formatted at 2026-09-27 02:43:05 on dist-test-slave-zv8z"
server_key: "b1509e70ace8696dd2763a3d758855fe"
server_key_iv: "ffa86ce0db7fe0d52752d150ab6cd2da"
server_key_version: "encryptionkey@0"
I20260927 02:43:05.507808 22224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476718023131-392-0/minicluster-data/master-0/wal/instance:
uuid: "9b9426a255b34f94bd4e61a45078315d"
format_stamp: "Formatted at 2026-09-27 02:43:05 on dist-test-slave-zv8z"
server_key: "b1509e70ace8696dd2763a3d758855fe"
server_key_iv: "ffa86ce0db7fe0d52752d150ab6cd2da"
server_key_version: "encryptionkey@0"
I20260927 02:43:05.511268 22224 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:43:05.513660 22239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:05.514910 22224 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:43:05.515028 22224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "9b9426a255b34f94bd4e61a45078315d"
format_stamp: "Formatted at 2026-09-27 02:43:05 on dist-test-slave-zv8z"
server_key: "b1509e70ace8696dd2763a3d758855fe"
server_key_iv: "ffa86ce0db7fe0d52752d150ab6cd2da"
server_key_version: "encryptionkey@0"
I20260927 02:43:05.515121 22224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:43:05.519412 22224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:43:05.522125 22224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:43:05.522308 22224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:43:05.528716 22224 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40397
I20260927 02:43:05.528748 22291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40397 every 8 connection(s)
I20260927 02:43:05.529559 22224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:43:05.532223 22292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:43:05.537494 22292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b9426a255b34f94bd4e61a45078315d: Bootstrap starting.
I20260927 02:43:05.538048 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 22224
I20260927 02:43:05.538194 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790476718023131-392-0/minicluster-data/master-0/wal/instance
I20260927 02:43:05.538483 392 external_mini_cluster.cc:1428] Setting key 9b7ab45a86c24347f85c10175fa27fd4
I20260927 02:43:05.540145 22292 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b9426a255b34f94bd4e61a45078315d: Neither blocks nor log segments found. Creating new log.
I20260927 02:43:05.540800 22292 log.cc:824] T 00000000000000000000000000000000 P 9b9426a255b34f94bd4e61a45078315d: Log is configured to *not* fsync() on all Append() calls
I20260927 02:43:05.542790 22292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b9426a255b34f94bd4e61a45078315d: No bootstrap required, opened a new log
I20260927 02:43:05.543208 22295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:05.539679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33479 (local address 127.0.98.62:40397)
0927 02:43:05.540046 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:05.540061 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:05.540096 (+ 35us) server_negotiation.cc:407] Connection header received
0927 02:43:05.540711 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:05.540729 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:05.541016 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:05.541333 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:05.542043 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:05.542048 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:05.542060 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:43:05.542068 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:05.542109 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:43:05.542127 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:05.542206 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:05.542358 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 02:43:05.542493 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":90,"threads_started":1}
I20260927 02:43:05.545805 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:43:05.546694 22292 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b9426a255b34f94bd4e61a45078315d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b9426a255b34f94bd4e61a45078315d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40397 } }
I20260927 02:43:05.546944 22292 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b9426a255b34f94bd4e61a45078315d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:43:05.547011 22292 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b9426a255b34f94bd4e61a45078315d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b9426a255b34f94bd4e61a45078315d, State: Initialized, Role: FOLLOWER
I20260927 02:43:05.547467 22292 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b9426a255b34f94bd4e61a45078315d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b9426a255b34f94bd4e61a45078315d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40397 } }
I20260927 02:43:05.547629 22292 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b9426a255b34f94bd4e61a45078315d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:43:05.547701 22292 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b9426a255b34f94bd4e61a45078315d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:43:05.547796 22292 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b9426a255b34f94bd4e61a45078315d [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:43:05.548591 22292 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b9426a255b34f94bd4e61a45078315d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b9426a255b34f94bd4e61a45078315d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40397 } }
I20260927 02:43:05.548887 22292 leader_election.cc:302] T 00000000000000000000000000000000 P 9b9426a255b34f94bd4e61a45078315d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b9426a255b34f94bd4e61a45078315d; no voters:
I20260927 02:43:05.549144 22292 leader_election.cc:288] T 00000000000000000000000000000000 P 9b9426a255b34f94bd4e61a45078315d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:43:05.549228 22302 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b9426a255b34f94bd4e61a45078315d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:43:05.549455 22295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:05.547937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39465 (local address 127.0.98.62:40397)
0927 02:43:05.548208 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:05.548212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:05.548254 (+ 42us) server_negotiation.cc:407] Connection header received
0927 02:43:05.548379 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:05.548382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:05.548408 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:05.548469 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:05.549078 (+ 609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:05.549081 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:05.549083 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:43:05.549094 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:05.549117 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:43:05.549128 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:05.549200 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:05.549323 (+ 123us) server_negotiation.cc:299] Negotiation successful
0927 02:43:05.549357 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205}
I20260927 02:43:05.549475 22302 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b9426a255b34f94bd4e61a45078315d [term 1 LEADER]: Becoming Leader. State: Replica: 9b9426a255b34f94bd4e61a45078315d, State: Running, Role: LEADER
I20260927 02:43:05.549831 22302 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b9426a255b34f94bd4e61a45078315d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b9426a255b34f94bd4e61a45078315d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40397 } }
I20260927 02:43:05.550370 22292 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b9426a255b34f94bd4e61a45078315d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:43:05.551342 22304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b9426a255b34f94bd4e61a45078315d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b9426a255b34f94bd4e61a45078315d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b9426a255b34f94bd4e61a45078315d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40397 } } }
I20260927 02:43:05.551455 22304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b9426a255b34f94bd4e61a45078315d [sys.catalog]: This master's current role is: LEADER
I20260927 02:43:05.551743 22303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b9426a255b34f94bd4e61a45078315d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b9426a255b34f94bd4e61a45078315d. Latest consensus state: current_term: 1 leader_uuid: "9b9426a255b34f94bd4e61a45078315d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b9426a255b34f94bd4e61a45078315d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40397 } } }
I20260927 02:43:05.551802 22311 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:43:05.551877 22303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b9426a255b34f94bd4e61a45078315d [sys.catalog]: This master's current role is: LEADER
I20260927 02:43:05.554579 22311 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:43:05.559940 22311 catalog_manager.cc:1347] Generated new cluster ID: 3727017f9be846399fff7fb0cf19bd95
I20260927 02:43:05.560007 22311 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:43:05.585055 22311 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:43:05.585415 22311 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:43:05.590835 22311 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b9426a255b34f94bd4e61a45078315d: Generated new TSK 0
I20260927 02:43:05.591622 22311 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:43:05.594053 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 22224
2026-09-27T02:43:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1264 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476718023131-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:43:05 dist-test-slave-zv8z krb5kdc[22323](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:43:05 dist-test-slave-zv8z krb5kdc[22323](info): set up 2 sockets
Sep 27 02:43:05 dist-test-slave-zv8z krb5kdc[22323](info): commencing operation
krb5kdc: starting...
W20260927 02:43:06.542166 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476718023131-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476718023131-392-0/krb5kdc/test-user.keytab.
Sep 27 02:43:06 dist-test-slave-zv8z krb5kdc[22323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:43:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:43:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:43:06.650674 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39589
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:43:06.753943 22336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:43:06.754189 22336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:43:06.754266 22336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:43:06.758147 22336 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:43:06.758201 22336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:43:06.758226 22336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:43:06.758263 22336 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:43:06.758287 22336 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:43:06.763285 22336 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:43:06.764197 22336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:43:06.765123 22336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:43:06.772915 22344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:06.772946 22341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:06.772956 22342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:43:06.773113 22336 server_base.cc:1034] running on GCE node
I20260927 02:43:06.773721 22336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:43:06.774808 22336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:43:06.775987 22336 hybrid_clock.cc:648] HybridClock initialized: now 1790476986775960 us; error 35 us; skew 500 ppm
Sep 27 02:43:06 dist-test-slave-zv8z krb5kdc[22323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:43:06.778553 22336 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:43:06.779325 22336 webserver.cc:460] Webserver started at http://127.0.98.62:43481/ using document root <none> and password file <none>
I20260927 02:43:06.779829 22336 fs_manager.cc:359] Metadata directory not provided
I20260927 02:43:06.779888 22336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:43:06.780077 22336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:43:06.781955 22336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476718023131-392-0/minicluster-data/master-0/data/instance:
uuid: "2505fbf6957841ee81fe4007f1a7350e"
format_stamp: "Formatted at 2026-09-27 02:43:06 on dist-test-slave-zv8z"
server_key: "ad5a03477c2a28987878452c634d0d85"
server_key_iv: "4f5cdd04f86bb2895a2afcfaa81efa85"
server_key_version: "encryptionkey@0"
I20260927 02:43:06.782476 22336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476718023131-392-0/minicluster-data/master-0/wal/instance:
uuid: "2505fbf6957841ee81fe4007f1a7350e"
format_stamp: "Formatted at 2026-09-27 02:43:06 on dist-test-slave-zv8z"
server_key: "ad5a03477c2a28987878452c634d0d85"
server_key_iv: "4f5cdd04f86bb2895a2afcfaa81efa85"
server_key_version: "encryptionkey@0"
I20260927 02:43:06.785842 22336 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260927 02:43:06.788098 22351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:06.789234 22336 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:43:06.789340 22336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "2505fbf6957841ee81fe4007f1a7350e"
format_stamp: "Formatted at 2026-09-27 02:43:06 on dist-test-slave-zv8z"
server_key: "ad5a03477c2a28987878452c634d0d85"
server_key_iv: "4f5cdd04f86bb2895a2afcfaa81efa85"
server_key_version: "encryptionkey@0"
I20260927 02:43:06.789428 22336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:43:06.812546 22336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:43:06.815572 22336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:43:06.815737 22336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:43:06.821947 22336 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39589
I20260927 02:43:06.821990 22403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39589 every 8 connection(s)
I20260927 02:43:06.822837 22336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:43:06.825500 22404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:43:06.826332 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 22336
I20260927 02:43:06.826468 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790476718023131-392-0/minicluster-data/master-0/wal/instance
I20260927 02:43:06.826787 392 external_mini_cluster.cc:1428] Setting key 8770296d560002b252526f06496727af
I20260927 02:43:06.831486 22404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2505fbf6957841ee81fe4007f1a7350e: Bootstrap starting.
I20260927 02:43:06.834014 22404 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2505fbf6957841ee81fe4007f1a7350e: Neither blocks nor log segments found. Creating new log.
I20260927 02:43:06.834906 22404 log.cc:824] T 00000000000000000000000000000000 P 2505fbf6957841ee81fe4007f1a7350e: Log is configured to *not* fsync() on all Append() calls
Sep 27 02:43:06 dist-test-slave-zv8z krb5kdc[22323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:43:06.836927 22404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2505fbf6957841ee81fe4007f1a7350e: No bootstrap required, opened a new log
I20260927 02:43:06.840539 22407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:06.827992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53095 (local address 127.0.98.62:39589)
0927 02:43:06.828371 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:06.828383 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:06.828418 (+ 35us) server_negotiation.cc:407] Connection header received
0927 02:43:06.829052 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:06.829068 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:06.829388 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:06.829646 (+ 258us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:06.830489 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:06.831453 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:06.832566 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:06.833016 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:06.833133 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:06.835751 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:06.835758 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:06.835774 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:06.835801 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:06.837566 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:06.837956 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:06.837962 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:06.837968 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:06.838096 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:06.838372 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:06.838375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:06.838377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:06.838921 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:06.839127 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:06.839319 (+ 192us) server_negotiation.cc:299] Negotiation successful
0927 02:43:06.839609 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":105,"threads_started":1}
I20260927 02:43:06.841176 22404 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2505fbf6957841ee81fe4007f1a7350e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2505fbf6957841ee81fe4007f1a7350e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39589 } }
I20260927 02:43:06.841379 22404 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2505fbf6957841ee81fe4007f1a7350e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:43:06.841414 22404 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2505fbf6957841ee81fe4007f1a7350e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2505fbf6957841ee81fe4007f1a7350e, State: Initialized, Role: FOLLOWER
I20260927 02:43:06.841800 22404 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2505fbf6957841ee81fe4007f1a7350e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2505fbf6957841ee81fe4007f1a7350e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39589 } }
I20260927 02:43:06.841907 22404 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2505fbf6957841ee81fe4007f1a7350e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:43:06.841943 22404 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2505fbf6957841ee81fe4007f1a7350e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:43:06.842002 22404 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2505fbf6957841ee81fe4007f1a7350e [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:43:06.842905 22404 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2505fbf6957841ee81fe4007f1a7350e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2505fbf6957841ee81fe4007f1a7350e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39589 } }
I20260927 02:43:06.843116 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:43:06.843281 22404 leader_election.cc:302] T 00000000000000000000000000000000 P 2505fbf6957841ee81fe4007f1a7350e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2505fbf6957841ee81fe4007f1a7350e; no voters:
I20260927 02:43:06.843554 22404 leader_election.cc:288] T 00000000000000000000000000000000 P 2505fbf6957841ee81fe4007f1a7350e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:43:06.843663 22409 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2505fbf6957841ee81fe4007f1a7350e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:43:06.843887 22409 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2505fbf6957841ee81fe4007f1a7350e [term 1 LEADER]: Becoming Leader. State: Replica: 2505fbf6957841ee81fe4007f1a7350e, State: Running, Role: LEADER
I20260927 02:43:06.844183 22409 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2505fbf6957841ee81fe4007f1a7350e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2505fbf6957841ee81fe4007f1a7350e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39589 } }
I20260927 02:43:06.844688 22404 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2505fbf6957841ee81fe4007f1a7350e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:43:06.845320 22409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2505fbf6957841ee81fe4007f1a7350e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2505fbf6957841ee81fe4007f1a7350e. Latest consensus state: current_term: 1 leader_uuid: "2505fbf6957841ee81fe4007f1a7350e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2505fbf6957841ee81fe4007f1a7350e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39589 } } }
I20260927 02:43:06.845430 22409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2505fbf6957841ee81fe4007f1a7350e [sys.catalog]: This master's current role is: LEADER
I20260927 02:43:06.845715 22410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2505fbf6957841ee81fe4007f1a7350e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2505fbf6957841ee81fe4007f1a7350e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2505fbf6957841ee81fe4007f1a7350e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39589 } } }
I20260927 02:43:06.845806 22410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2505fbf6957841ee81fe4007f1a7350e [sys.catalog]: This master's current role is: LEADER
W20260927 02:43:06.849334 22429 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2505fbf6957841ee81fe4007f1a7350e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:43:06.849406 22429 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:43:06.849483 22422 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:43:06.850354 22422 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:43:06.853823 22407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:06.845114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54387 (local address 127.0.98.62:39589)
0927 02:43:06.845206 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:06.845209 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:06.845469 (+ 260us) server_negotiation.cc:407] Connection header received
0927 02:43:06.845667 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:06.845671 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:06.845717 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:06.845795 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:06.846643 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:06.847330 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:06.848375 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:06.848712 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:06.848829 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:06.850422 (+ 1593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:06.850426 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:06.850428 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:06.850453 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:06.852054 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:06.852492 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:06.852495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:06.852497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:06.852583 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:06.852869 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:06.852873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:06.852875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:06.853099 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:06.853226 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:06.853538 (+ 312us) server_negotiation.cc:299] Negotiation successful
0927 02:43:06.853672 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 02:43:06.855613 22422 catalog_manager.cc:1347] Generated new cluster ID: ab95bc3def1b4ca9b180014e6b214f39
I20260927 02:43:06.855667 22422 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:43:06.860234 22422 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:43:06.861326 22422 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:43:06.873803 22422 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2505fbf6957841ee81fe4007f1a7350e: Generated new TSK 0
I20260927 02:43:06.874683 22422 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:43:06.884750 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 22336
2026-09-27T02:43:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1290 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476718023131-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:43:06 dist-test-slave-zv8z krb5kdc[22435](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:43:06 dist-test-slave-zv8z krb5kdc[22435](info): set up 2 sockets
Sep 27 02:43:06 dist-test-slave-zv8z krb5kdc[22435](info): commencing operation
krb5kdc: starting...
W20260927 02:43:07.826309 392 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476718023131-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476718023131-392-0/krb5kdc/test-user.keytab.
Sep 27 02:43:07 dist-test-slave-zv8z krb5kdc[22435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:43:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:43:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:43:07.936440 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37043
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37043
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:43:08.040709 22448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:43:08.041013 22448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:43:08.041081 22448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:43:08.045017 22448 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:43:08.045075 22448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:43:08.045101 22448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:43:08.045125 22448 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:43:08.045152 22448 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:43:08.050190 22448 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37043
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:43:08.051152 22448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:43:08.052095 22448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:43:08.060111 22454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:08.060089 22456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:08.060089 22453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:43:08.060420 22448 server_base.cc:1034] running on GCE node
I20260927 02:43:08.060927 22448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:43:08.061959 22448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:43:08.063124 22448 hybrid_clock.cc:648] HybridClock initialized: now 1790476988063110 us; error 25 us; skew 500 ppm
Sep 27 02:43:08 dist-test-slave-zv8z krb5kdc[22435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476988, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:43:08.069018 22448 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:43:08.069888 22448 webserver.cc:460] Webserver started at http://127.0.98.62:41443/ using document root <none> and password file <none>
I20260927 02:43:08.070466 22448 fs_manager.cc:359] Metadata directory not provided
I20260927 02:43:08.070534 22448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:43:08.070748 22448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:43:08.072755 22448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476718023131-392-0/minicluster-data/master-0/data/instance:
uuid: "aefe7e9471c5421c97761fc017fb0be5"
format_stamp: "Formatted at 2026-09-27 02:43:08 on dist-test-slave-zv8z"
server_key: "94fadd0597b90c4070cfd86ab388acc0"
server_key_iv: "7b5b263f457f479c7fbb5c0e59388890"
server_key_version: "encryptionkey@0"
I20260927 02:43:08.073267 22448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476718023131-392-0/minicluster-data/master-0/wal/instance:
uuid: "aefe7e9471c5421c97761fc017fb0be5"
format_stamp: "Formatted at 2026-09-27 02:43:08 on dist-test-slave-zv8z"
server_key: "94fadd0597b90c4070cfd86ab388acc0"
server_key_iv: "7b5b263f457f479c7fbb5c0e59388890"
server_key_version: "encryptionkey@0"
I20260927 02:43:08.076843 22448 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 02:43:08.079280 22463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:08.080412 22448 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:43:08.080530 22448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "aefe7e9471c5421c97761fc017fb0be5"
format_stamp: "Formatted at 2026-09-27 02:43:08 on dist-test-slave-zv8z"
server_key: "94fadd0597b90c4070cfd86ab388acc0"
server_key_iv: "7b5b263f457f479c7fbb5c0e59388890"
server_key_version: "encryptionkey@0"
I20260927 02:43:08.080619 22448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:43:08.126377 22448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:43:08.129539 22448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:43:08.129714 22448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:43:08.136178 22448 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37043
I20260927 02:43:08.136224 22515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37043 every 8 connection(s)
I20260927 02:43:08.137089 22448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:43:08.139863 22516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:43:08.142160 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 22448
I20260927 02:43:08.142341 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790476718023131-392-0/minicluster-data/master-0/wal/instance
I20260927 02:43:08.142639 392 external_mini_cluster.cc:1428] Setting key bed0f72fbd93266a5ae5f24099a286ea
I20260927 02:43:08.145987 22516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aefe7e9471c5421c97761fc017fb0be5: Bootstrap starting.
I20260927 02:43:08.148389 22516 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aefe7e9471c5421c97761fc017fb0be5: Neither blocks nor log segments found. Creating new log.
I20260927 02:43:08.149206 22516 log.cc:824] T 00000000000000000000000000000000 P aefe7e9471c5421c97761fc017fb0be5: Log is configured to *not* fsync() on all Append() calls
I20260927 02:43:08.151223 22516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aefe7e9471c5421c97761fc017fb0be5: No bootstrap required, opened a new log
I20260927 02:43:08.151530 22519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:08.143948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41449 (local address 127.0.98.62:37043)
0927 02:43:08.144339 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:08.144355 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:08.144392 (+ 37us) server_negotiation.cc:407] Connection header received
0927 02:43:08.145057 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:08.145075 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:08.145410 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:08.145682 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:08.146006 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:08.147033 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:08.148166 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:08.148658 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:08.148776 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:08.149674 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:08.149681 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:08.149696 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:43:08.150183 (+ 487us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:08.150289 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:43:08.150319 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:08.150397 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:08.150531 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 02:43:08.150781 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":105,"threads_started":1}
I20260927 02:43:08.154415 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:43:08.155092 22516 raft_consensus.cc:348] T 00000000000000000000000000000000 P aefe7e9471c5421c97761fc017fb0be5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aefe7e9471c5421c97761fc017fb0be5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37043 } }
I20260927 02:43:08.155314 22516 raft_consensus.cc:374] T 00000000000000000000000000000000 P aefe7e9471c5421c97761fc017fb0be5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:43:08.155369 22516 raft_consensus.cc:729] T 00000000000000000000000000000000 P aefe7e9471c5421c97761fc017fb0be5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe7e9471c5421c97761fc017fb0be5, State: Initialized, Role: FOLLOWER
I20260927 02:43:08.155797 22516 consensus_queue.cc:260] T 00000000000000000000000000000000 P aefe7e9471c5421c97761fc017fb0be5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aefe7e9471c5421c97761fc017fb0be5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37043 } }
I20260927 02:43:08.155934 22516 raft_consensus.cc:388] T 00000000000000000000000000000000 P aefe7e9471c5421c97761fc017fb0be5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:43:08.156001 22516 raft_consensus.cc:482] T 00000000000000000000000000000000 P aefe7e9471c5421c97761fc017fb0be5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:43:08.156097 22516 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aefe7e9471c5421c97761fc017fb0be5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:43:08.157148 22516 raft_consensus.cc:504] T 00000000000000000000000000000000 P aefe7e9471c5421c97761fc017fb0be5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aefe7e9471c5421c97761fc017fb0be5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37043 } }
I20260927 02:43:08.157454 22516 leader_election.cc:302] T 00000000000000000000000000000000 P aefe7e9471c5421c97761fc017fb0be5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aefe7e9471c5421c97761fc017fb0be5; no voters:
I20260927 02:43:08.157727 22516 leader_election.cc:288] T 00000000000000000000000000000000 P aefe7e9471c5421c97761fc017fb0be5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:43:08.157804 22526 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aefe7e9471c5421c97761fc017fb0be5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:43:08.158017 22526 raft_consensus.cc:686] T 00000000000000000000000000000000 P aefe7e9471c5421c97761fc017fb0be5 [term 1 LEADER]: Becoming Leader. State: Replica: aefe7e9471c5421c97761fc017fb0be5, State: Running, Role: LEADER
I20260927 02:43:08.158380 22526 consensus_queue.cc:237] T 00000000000000000000000000000000 P aefe7e9471c5421c97761fc017fb0be5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aefe7e9471c5421c97761fc017fb0be5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37043 } }
I20260927 02:43:08.158973 22516 sys_catalog.cc:564] T 00000000000000000000000000000000 P aefe7e9471c5421c97761fc017fb0be5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:43:08.159538 22526 sys_catalog.cc:455] T 00000000000000000000000000000000 P aefe7e9471c5421c97761fc017fb0be5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aefe7e9471c5421c97761fc017fb0be5. Latest consensus state: current_term: 1 leader_uuid: "aefe7e9471c5421c97761fc017fb0be5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aefe7e9471c5421c97761fc017fb0be5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37043 } } }
I20260927 02:43:08.159714 22526 sys_catalog.cc:458] T 00000000000000000000000000000000 P aefe7e9471c5421c97761fc017fb0be5 [sys.catalog]: This master's current role is: LEADER
I20260927 02:43:08.160123 22535 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:43:08.161024 22519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:08.156285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52059 (local address 127.0.98.62:37043)
0927 02:43:08.156380 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:08.156384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:08.156581 (+ 197us) server_negotiation.cc:407] Connection header received
0927 02:43:08.156698 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:08.156699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:08.156732 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:08.156792 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:08.157047 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:08.157593 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:08.158666 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:08.158985 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:08.159097 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:08.160589 (+ 1492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:08.160593 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:08.160595 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:43:08.160607 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:08.160633 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:43:08.160645 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:08.160690 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:08.160848 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 02:43:08.160901 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 02:43:08.160097 22527 sys_catalog.cc:455] T 00000000000000000000000000000000 P aefe7e9471c5421c97761fc017fb0be5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aefe7e9471c5421c97761fc017fb0be5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aefe7e9471c5421c97761fc017fb0be5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37043 } } }
I20260927 02:43:08.161298 22527 sys_catalog.cc:458] T 00000000000000000000000000000000 P aefe7e9471c5421c97761fc017fb0be5 [sys.catalog]: This master's current role is: LEADER
I20260927 02:43:08.163913 22535 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:43:08.169054 22535 catalog_manager.cc:1347] Generated new cluster ID: 0af3c96ba9f4445cb97540aff5ae1d08
I20260927 02:43:08.169117 22535 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:43:08.205024 22535 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:43:08.206182 22535 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:43:08.215245 22535 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aefe7e9471c5421c97761fc017fb0be5: Generated new TSK 0
I20260927 02:43:08.215986 22535 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:43:08.223456 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 22448
2026-09-27T02:43:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1339 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476718023131-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:43:08 dist-test-slave-zv8z krb5kdc[22547](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:43:08 dist-test-slave-zv8z krb5kdc[22547](info): set up 2 sockets
Sep 27 02:43:08 dist-test-slave-zv8z krb5kdc[22547](info): commencing operation
krb5kdc: starting...
W20260927 02:43:09.170332 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476718023131-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476718023131-392-0/krb5kdc/test-user.keytab.
Sep 27 02:43:09 dist-test-slave-zv8z krb5kdc[22547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:43:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:43:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:43:09.280853 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42061
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:43:09.386034 22560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:43:09.386353 22560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:43:09.386420 22560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:43:09.390355 22560 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:43:09.390413 22560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:43:09.390430 22560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:43:09.390446 22560 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:43:09.390472 22560 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:43:09.395643 22560 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:43:09.396613 22560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:43:09.397581 22560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:43:09.405706 22566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:09.405714 22568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:09.405714 22565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:43:09.406112 22560 server_base.cc:1034] running on GCE node
I20260927 02:43:09.406668 22560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:43:09.407737 22560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:43:09.408905 22560 hybrid_clock.cc:648] HybridClock initialized: now 1790476989408879 us; error 36 us; skew 500 ppm
Sep 27 02:43:09 dist-test-slave-zv8z krb5kdc[22547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:43:09.411510 22560 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:43:09.412360 22560 webserver.cc:460] Webserver started at http://127.0.98.62:35481/ using document root <none> and password file <none>
I20260927 02:43:09.412863 22560 fs_manager.cc:359] Metadata directory not provided
I20260927 02:43:09.412925 22560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:43:09.413101 22560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:43:09.415091 22560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476718023131-392-0/minicluster-data/master-0/data/instance:
uuid: "9a5bd606880946dbbde206ac35625ec7"
format_stamp: "Formatted at 2026-09-27 02:43:09 on dist-test-slave-zv8z"
server_key: "39f86100c2af6c4985780654ce789ef7"
server_key_iv: "a00783017ed6c82ad80329c93da8a775"
server_key_version: "encryptionkey@0"
I20260927 02:43:09.415609 22560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476718023131-392-0/minicluster-data/master-0/wal/instance:
uuid: "9a5bd606880946dbbde206ac35625ec7"
format_stamp: "Formatted at 2026-09-27 02:43:09 on dist-test-slave-zv8z"
server_key: "39f86100c2af6c4985780654ce789ef7"
server_key_iv: "a00783017ed6c82ad80329c93da8a775"
server_key_version: "encryptionkey@0"
I20260927 02:43:09.419070 22560 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:43:09.421479 22575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:09.422773 22560 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:43:09.422870 22560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "9a5bd606880946dbbde206ac35625ec7"
format_stamp: "Formatted at 2026-09-27 02:43:09 on dist-test-slave-zv8z"
server_key: "39f86100c2af6c4985780654ce789ef7"
server_key_iv: "a00783017ed6c82ad80329c93da8a775"
server_key_version: "encryptionkey@0"
I20260927 02:43:09.422950 22560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:43:09.427109 22560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:43:09.429795 22560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:43:09.429932 22560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:43:09.436232 22560 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42061
I20260927 02:43:09.436264 22627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42061 every 8 connection(s)
I20260927 02:43:09.437074 22560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:43:09.439646 22628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:43:09.444768 22628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a5bd606880946dbbde206ac35625ec7: Bootstrap starting.
I20260927 02:43:09.445935 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 22560
I20260927 02:43:09.446053 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790476718023131-392-0/minicluster-data/master-0/wal/instance
I20260927 02:43:09.446348 392 external_mini_cluster.cc:1428] Setting key 13d24b2ae8854663af522c7ee452b4dd
I20260927 02:43:09.447309 22628 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a5bd606880946dbbde206ac35625ec7: Neither blocks nor log segments found. Creating new log.
I20260927 02:43:09.447997 22628 log.cc:824] T 00000000000000000000000000000000 P 9a5bd606880946dbbde206ac35625ec7: Log is configured to *not* fsync() on all Append() calls
I20260927 02:43:09.449838 22628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a5bd606880946dbbde206ac35625ec7: No bootstrap required, opened a new log
I20260927 02:43:09.451062 22631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:09.447378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51807 (local address 127.0.98.62:42061)
0927 02:43:09.447806 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:09.447820 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:09.447854 (+ 34us) server_negotiation.cc:407] Connection header received
0927 02:43:09.448505 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:09.448522 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:09.448803 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:09.449092 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:09.449874 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:09.449879 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:09.449890 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:43:09.449900 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:09.449970 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:43:09.449992 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:09.450076 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:09.450222 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 02:43:09.450393 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":140,"threads_started":1}
I20260927 02:43:09.453359 22628 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a5bd606880946dbbde206ac35625ec7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5bd606880946dbbde206ac35625ec7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42061 } }
I20260927 02:43:09.453559 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:43:09.453594 22628 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a5bd606880946dbbde206ac35625ec7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:43:09.453644 22628 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a5bd606880946dbbde206ac35625ec7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a5bd606880946dbbde206ac35625ec7, State: Initialized, Role: FOLLOWER
I20260927 02:43:09.454064 22628 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a5bd606880946dbbde206ac35625ec7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5bd606880946dbbde206ac35625ec7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42061 } }
I20260927 02:43:09.454214 22628 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a5bd606880946dbbde206ac35625ec7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:43:09.454288 22628 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a5bd606880946dbbde206ac35625ec7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:43:09.454382 22628 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a5bd606880946dbbde206ac35625ec7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:43:09.455442 22628 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a5bd606880946dbbde206ac35625ec7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5bd606880946dbbde206ac35625ec7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42061 } }
I20260927 02:43:09.455772 22628 leader_election.cc:302] T 00000000000000000000000000000000 P 9a5bd606880946dbbde206ac35625ec7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a5bd606880946dbbde206ac35625ec7; no voters:
I20260927 02:43:09.456091 22628 leader_election.cc:288] T 00000000000000000000000000000000 P 9a5bd606880946dbbde206ac35625ec7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:43:09.456173 22638 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a5bd606880946dbbde206ac35625ec7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:43:09.456389 22638 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a5bd606880946dbbde206ac35625ec7 [term 1 LEADER]: Becoming Leader. State: Replica: 9a5bd606880946dbbde206ac35625ec7, State: Running, Role: LEADER
I20260927 02:43:09.457012 22638 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a5bd606880946dbbde206ac35625ec7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5bd606880946dbbde206ac35625ec7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42061 } }
I20260927 02:43:09.457317 22628 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a5bd606880946dbbde206ac35625ec7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:43:09.457360 22631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:09.455655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41197 (local address 127.0.98.62:42061)
0927 02:43:09.455767 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:09.455771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:09.455944 (+ 173us) server_negotiation.cc:407] Connection header received
0927 02:43:09.456082 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:09.456085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:09.456119 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:09.456188 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:09.456932 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:09.456935 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:09.456938 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:43:09.456948 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:09.456974 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:43:09.456987 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:09.457039 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:09.457204 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 02:43:09.457254 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 02:43:09.458312 22638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a5bd606880946dbbde206ac35625ec7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a5bd606880946dbbde206ac35625ec7. Latest consensus state: current_term: 1 leader_uuid: "9a5bd606880946dbbde206ac35625ec7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5bd606880946dbbde206ac35625ec7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42061 } } }
I20260927 02:43:09.458472 22638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a5bd606880946dbbde206ac35625ec7 [sys.catalog]: This master's current role is: LEADER
I20260927 02:43:09.458843 22647 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:43:09.458846 22639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a5bd606880946dbbde206ac35625ec7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a5bd606880946dbbde206ac35625ec7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5bd606880946dbbde206ac35625ec7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42061 } } }
I20260927 02:43:09.458957 22639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a5bd606880946dbbde206ac35625ec7 [sys.catalog]: This master's current role is: LEADER
I20260927 02:43:09.461813 22647 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:43:09.466781 22647 catalog_manager.cc:1347] Generated new cluster ID: 45cf356e8462455c8216c0d0f58c8e40
I20260927 02:43:09.466841 22647 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:43:09.492658 22647 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:43:09.492965 22647 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:43:09.500780 22647 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a5bd606880946dbbde206ac35625ec7: Generated new TSK 0
I20260927 02:43:09.501410 22647 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:43:09.508493 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 22560
2026-09-27T02:43:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1284 ms)
[----------] 18 tests from AuthTokenIssuingTest (23314 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:43:09 dist-test-slave-zv8z krb5kdc[22659](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:43:09 dist-test-slave-zv8z krb5kdc[22659](info): set up 2 sockets
Sep 27 02:43:09 dist-test-slave-zv8z krb5kdc[22659](info): commencing operation
krb5kdc: starting...
W20260927 02:43:10.458317 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/krb5kdc/test-user.keytab.
Sep 27 02:43:10 dist-test-slave-zv8z krb5kdc[22659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:43:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:43:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:43:10.585188 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37273
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37273,127.0.98.61:43741,127.0.98.60:43471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:43:10.690593 22672 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 02:43:10.690863 22672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:43:10.690894 22672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:43:10.690953 22672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:43:10.694921 22672 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:43:10.694974 22672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:43:10.695000 22672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:43:10.695020 22672 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:43:10.695045 22672 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:43:10.700119 22672 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37273,127.0.98.61:43741,127.0.98.60:43471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:43:10.701097 22672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:43:10.702064 22672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:43:10.710106 22678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:10.710115 22680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:10.710115 22677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:43:10.710466 22672 server_base.cc:1034] running on GCE node
I20260927 02:43:10.711122 22672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:43:10.712173 22672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:43:10.713331 22672 hybrid_clock.cc:648] HybridClock initialized: now 1790476990713310 us; error 35 us; skew 500 ppm
Sep 27 02:43:10 dist-test-slave-zv8z krb5kdc[22659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:43:10.715862 22672 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:43:10.716722 22672 webserver.cc:460] Webserver started at http://127.0.98.62:32899/ using document root <none> and password file <none>
I20260927 02:43:10.717237 22672 fs_manager.cc:359] Metadata directory not provided
I20260927 02:43:10.717299 22672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:43:10.717476 22672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:43:10.719431 22672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/minicluster-data/master-0/data/instance:
uuid: "978d68c4839d4229ae8256d05b55e015"
format_stamp: "Formatted at 2026-09-27 02:43:10 on dist-test-slave-zv8z"
server_key: "fe1f24913b0ba3c582491425a362480f"
server_key_iv: "8166394106840728ce4a7a8241e9b5d6"
server_key_version: "encryptionkey@0"
I20260927 02:43:10.719944 22672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/minicluster-data/master-0/wal/instance:
uuid: "978d68c4839d4229ae8256d05b55e015"
format_stamp: "Formatted at 2026-09-27 02:43:10 on dist-test-slave-zv8z"
server_key: "fe1f24913b0ba3c582491425a362480f"
server_key_iv: "8166394106840728ce4a7a8241e9b5d6"
server_key_version: "encryptionkey@0"
I20260927 02:43:10.723446 22672 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 02:43:10.725665 22687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:10.726907 22672 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:43:10.727054 22672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "978d68c4839d4229ae8256d05b55e015"
format_stamp: "Formatted at 2026-09-27 02:43:10 on dist-test-slave-zv8z"
server_key: "fe1f24913b0ba3c582491425a362480f"
server_key_iv: "8166394106840728ce4a7a8241e9b5d6"
server_key_version: "encryptionkey@0"
I20260927 02:43:10.727146 22672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:43:10.743234 22672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:43:10.746119 22672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:43:10.746325 22672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:43:10.752826 22672 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37273
I20260927 02:43:10.753072 22739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37273 every 8 connection(s)
I20260927 02:43:10.753738 22672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:43:10.756443 22740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:43:10.760015 22740 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37273 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:43:10.760874 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 22672
I20260927 02:43:10.760990 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/minicluster-data/master-0/wal/instance
I20260927 02:43:10.761241 392 external_mini_cluster.cc:1428] Setting key d4350ebb112189efa8633e0f89486225
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 27 02:43:10 dist-test-slave-zv8z krb5kdc[22659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:43:10.776419 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:10.762202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37273 (local address 127.0.0.1:46492)
0927 02:43:10.762834 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:10.762879 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:10.763648 (+ 769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:10.764352 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:10.764356 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:10.764670 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:43:10.765026 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:10.765040 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:10.766691 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:10.766694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:10.767436 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:10.767444 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:10.768238 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:10.768240 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:10.768329 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:10.768906 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:43:10.768921 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:43:10.771967 (+ 3046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:43:10.773694 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:10.773703 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:10.773714 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:10.773984 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:10.774286 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:10.774290 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:10.774292 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:10.774424 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:10.775005 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:43:10.775021 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:43:10.775183 (+ 162us) client_negotiation.cc:769] Sending connection context
0927 02:43:10.775363 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 02:43:10.775748 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"mutex_wait_us":109,"spinlock_wait_cycles":71936,"thread_start_us":133,"threads_started":1}
I20260927 02:43:10.776417 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:10.762202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46492 (local address 127.0.98.62:37273)
0927 02:43:10.762845 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:10.762864 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:10.762981 (+ 117us) server_negotiation.cc:407] Connection header received
0927 02:43:10.763939 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:10.763944 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:10.764166 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:10.764375 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:10.765163 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:10.766577 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:10.767548 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:10.768101 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:10.768294 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:10.772102 (+ 3808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:10.772107 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:10.772117 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:10.772137 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:10.773596 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:10.774080 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:10.774085 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:10.774086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:10.774186 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:10.774500 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:10.774504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:10.774505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:10.774893 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:10.775020 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:10.775422 (+ 402us) server_negotiation.cc:299] Negotiation successful
0927 02:43:10.775653 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":82,"server-negotiator.queue_time_us":504,"spinlock_wait_cycles":6912,"thread_start_us":133,"threads_started":1}
I20260927 02:43:10.779204 22740 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43741 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:43:10.779882 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:10.779637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43741 (local address 127.0.0.1:42422)
0927 02:43:10.779748 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:10.779819 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 02:43:10.780087 22689 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43741: connect: Connection refused (error 111)
I20260927 02:43:10.780959 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:10.780749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43741 (local address 127.0.0.1:42436)
0927 02:43:10.780860 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:10.780907 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 02:43:10.781077 22740 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43741: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43741: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 02:43:10.786753 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43741
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37273,127.0.98.61:43741,127.0.98.60:43471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:43:10.830348 22740 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43741 } attempt: 1
I20260927 02:43:10.831300 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:10.830954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43741 (local address 127.0.0.1:42438)
0927 02:43:10.831139 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:10.831210 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 02:43:10.832034 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:10.831844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43741 (local address 127.0.0.1:42446)
0927 02:43:10.831934 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:10.831979 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 02:43:10.832141 22740 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43741: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43741: connect: Connection refused (error 111)
W20260927 02:43:10.891953 22747 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 02:43:10.892206 22747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:43:10.892227 22747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:43:10.892262 22747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:43:10.896234 22747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:43:10.896286 22747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:43:10.896304 22747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:43:10.896318 22747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:43:10.896332 22747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:43:10.900318 22740 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43741 } attempt: 2
I20260927 02:43:10.901108 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:10.900791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43741 (local address 127.0.0.1:42450)
0927 02:43:10.900976 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:10.901034 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 02:43:10.901484 22747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37273,127.0.98.61:43741,127.0.98.60:43471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:43:10.901837 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:10.901629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43741 (local address 127.0.0.1:42456)
0927 02:43:10.901723 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:10.901771 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 02:43:10.902026 22740 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43741: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43741: connect: Connection refused (error 111)
I20260927 02:43:10.902586 22747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:43:10.903523 22747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:43:10.911427 22755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:10.911433 22753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:10.911440 22752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:43:10.911593 22747 server_base.cc:1034] running on GCE node
I20260927 02:43:10.912176 22747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:43:10.913245 22747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:43:10.914391 22747 hybrid_clock.cc:648] HybridClock initialized: now 1790476990914371 us; error 32 us; skew 500 ppm
Sep 27 02:43:10 dist-test-slave-zv8z krb5kdc[22659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:43:10.916986 22747 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 02:43:10.917800 22747 webserver.cc:460] Webserver started at http://127.0.98.61:46847/ using document root <none> and password file <none>
I20260927 02:43:10.918319 22747 fs_manager.cc:359] Metadata directory not provided
I20260927 02:43:10.918383 22747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:43:10.918543 22747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:43:10.920410 22747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/minicluster-data/master-1/data/instance:
uuid: "46989b41783443839c9fabb12ea57d73"
format_stamp: "Formatted at 2026-09-27 02:43:10 on dist-test-slave-zv8z"
server_key: "f443ffe4a840d75f9dfa3f2e9e656189"
server_key_iv: "082fdeb79114d34bf146e8791a6d8fb4"
server_key_version: "encryptionkey@0"
I20260927 02:43:10.920892 22747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/minicluster-data/master-1/wal/instance:
uuid: "46989b41783443839c9fabb12ea57d73"
format_stamp: "Formatted at 2026-09-27 02:43:10 on dist-test-slave-zv8z"
server_key: "f443ffe4a840d75f9dfa3f2e9e656189"
server_key_iv: "082fdeb79114d34bf146e8791a6d8fb4"
server_key_version: "encryptionkey@0"
I20260927 02:43:10.924410 22747 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 02:43:10.926717 22762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:10.927889 22747 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:43:10.927987 22747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/minicluster-data/master-1/wal
uuid: "46989b41783443839c9fabb12ea57d73"
format_stamp: "Formatted at 2026-09-27 02:43:10 on dist-test-slave-zv8z"
server_key: "f443ffe4a840d75f9dfa3f2e9e656189"
server_key_iv: "082fdeb79114d34bf146e8791a6d8fb4"
server_key_version: "encryptionkey@0"
I20260927 02:43:10.928067 22747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:43:10.947779 22747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:43:10.950922 22747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:43:10.951071 22747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:43:10.957644 22747 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43741
I20260927 02:43:10.957895 22814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43741 every 8 connection(s)
I20260927 02:43:10.958601 22747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/minicluster-data/master-1/data/info.pb
I20260927 02:43:10.961227 22815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:43:10.962407 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 22747
I20260927 02:43:10.962532 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/minicluster-data/master-1/wal/instance
I20260927 02:43:10.962814 392 external_mini_cluster.cc:1428] Setting key de69d5ce826afd75b7d01504b44f4ba3
I20260927 02:43:10.965438 22815 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37273 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 27 02:43:10 dist-test-slave-zv8z krb5kdc[22659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:43:10.980744 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:10.967536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46498 (local address 127.0.98.62:37273)
0927 02:43:10.967659 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:10.967663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:10.968766 (+ 1103us) server_negotiation.cc:407] Connection header received
0927 02:43:10.969689 (+ 923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:10.969692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:10.969751 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:10.969821 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:10.971066 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:10.971593 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:10.972706 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:10.972968 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:10.973059 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:10.976863 (+ 3804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:10.976867 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:10.976870 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:10.976899 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:10.978564 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:10.979115 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:10.979120 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:10.979122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:10.979218 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:10.979541 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:10.979545 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:10.979547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:10.979787 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:10.979889 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:10.980431 (+ 542us) server_negotiation.cc:299] Negotiation successful
0927 02:43:10.980568 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 02:43:10.981575 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:10.968114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37273 (local address 127.0.0.1:46498)
0927 02:43:10.968616 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:10.968658 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:10.969486 (+ 828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:10.970035 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:10.970042 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:10.970423 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:43:10.970923 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:10.970935 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:10.971710 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:10.971712 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:10.972590 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:10.972600 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:10.974170 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:10.974173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:10.974357 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:10.974975 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:43:10.974992 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:43:10.976663 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:43:10.978706 (+ 2043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:10.978712 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:10.978726 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:10.979007 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:10.979304 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:10.979307 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:10.979308 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:10.979454 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:10.979906 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:43:10.979911 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:43:10.980178 (+ 267us) client_negotiation.cc:769] Sending connection context
0927 02:43:10.980341 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 02:43:10.980632 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":135,"threads_started":1}
I20260927 02:43:10.983227 22815 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43741 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 02:43:10.987600 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43471
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37273,127.0.98.61:43741,127.0.98.60:43471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 02:43:10 dist-test-slave-zv8z krb5kdc[22659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 02:43:10.992019 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:10.983803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43741 (local address 127.0.0.1:42472)
0927 02:43:10.983920 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:10.983931 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:10.983996 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:10.984534 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:10.984536 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:10.984696 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:43:10.984779 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:10.984784 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:10.985803 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:10.985806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:10.986369 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:10.986378 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:10.986979 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:10.986982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:10.987072 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:10.987543 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:43:10.987553 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:43:10.988885 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:43:10.990641 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:10.990644 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:10.990646 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:10.990818 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:10.991066 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:10.991069 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:10.991071 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:10.991142 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:10.991548 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:43:10.991551 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:43:10.991634 (+ 83us) client_negotiation.cc:769] Sending connection context
0927 02:43:10.991718 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 02:43:10.991847 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 02:43:10.992115 22820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:10.983885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42472 (local address 127.0.98.61:43741)
0927 02:43:10.984205 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:10.984213 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:10.984231 (+ 18us) server_negotiation.cc:407] Connection header received
0927 02:43:10.984263 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:10.984267 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:10.984419 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:10.984543 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:10.984900 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:10.985705 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:10.986479 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:10.986882 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:10.986971 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:10.989015 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:10.989024 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:10.989035 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:10.989054 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:10.990549 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:10.990909 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:10.990914 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:10.990915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:10.990982 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:10.991213 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:10.991216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:10.991218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:10.991450 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:10.991565 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:10.991806 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 02:43:10.991927 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":140,"threads_started":1}
I20260927 02:43:10.993209 22740 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43741 } attempt: 3
I20260927 02:43:10.994391 22815 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43471 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:43:10.995157 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:10.994864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43471 (local address 127.0.0.1:58118)
0927 02:43:10.994993 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:10.995079 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 02:43:10.995417 22763 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43471: connect: Connection refused (error 111)
I20260927 02:43:10.996338 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:10.996119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43471 (local address 127.0.0.1:58126)
0927 02:43:10.996223 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:10.996274 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 02:43:10.996572 22815 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43471: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43471: connect: Connection refused (error 111)
Sep 27 02:43:10 dist-test-slave-zv8z krb5kdc[22659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 02:43:11.001991 22820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:10.994106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42478 (local address 127.0.98.61:43741)
0927 02:43:10.994221 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:10.994225 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:10.994236 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:43:10.994288 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:10.994291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:10.994330 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:10.994435 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:10.994836 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:10.995465 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:10.996902 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:10.997133 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:10.997212 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:10.999310 (+ 2098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:10.999313 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:10.999315 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:10.999335 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:11.000549 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:11.000989 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:11.000991 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:11.000992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:11.001064 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:11.001318 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:11.001321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:11.001322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:11.001489 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:11.001566 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:11.001796 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 02:43:11.001885 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 02:43:11.002063 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:10.993648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43741 (local address 127.0.0.1:42478)
0927 02:43:10.993886 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:10.993901 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:10.993976 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:10.994460 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:10.994463 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:10.994612 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:43:10.994702 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:10.994708 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:10.995600 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:10.995603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:10.996788 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:10.996798 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:10.997230 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:10.997233 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:10.997332 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:10.997779 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:43:10.997790 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:43:10.999188 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:43:11.000643 (+ 1455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:11.000647 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:11.000649 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:11.000892 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:11.001153 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:11.001156 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:11.001158 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:11.001241 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:11.001567 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:43:11.001570 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:43:11.001665 (+ 95us) client_negotiation.cc:769] Sending connection context
0927 02:43:11.001762 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 02:43:11.001893 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164}
I20260927 02:43:11.003239 22740 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43471 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:43:11.003894 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:11.003665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43471 (local address 127.0.0.1:58142)
0927 02:43:11.003775 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:11.003827 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 02:43:11.004530 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:11.004298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43471 (local address 127.0.0.1:58154)
0927 02:43:11.004432 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:11.004478 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260927 02:43:11.004634 22740 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43471: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43471: connect: Connection refused (error 111)
I20260927 02:43:11.035785 22740 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43471 } attempt: 1
I20260927 02:43:11.036602 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:11.036282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43471 (local address 127.0.0.1:58158)
0927 02:43:11.036438 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:11.036512 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 02:43:11.037328 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:11.037114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43471 (local address 127.0.0.1:58172)
0927 02:43:11.037211 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:11.037260 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 02:43:11.037528 22740 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43471: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43471: connect: Connection refused (error 111)
I20260927 02:43:11.045841 22815 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43471 } attempt: 1
I20260927 02:43:11.046625 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:11.046362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43471 (local address 127.0.0.1:58176)
0927 02:43:11.046494 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:11.046552 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 02:43:11.047410 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:11.047167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43471 (local address 127.0.0.1:58188)
0927 02:43:11.047300 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:11.047348 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260927 02:43:11.047539 22815 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43471: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43471: connect: Connection refused (error 111)
W20260927 02:43:11.097508 22821 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 02:43:11.097775 22821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:43:11.097796 22821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:43:11.097842 22821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:43:11.101850 22821 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:43:11.101912 22821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:43:11.101930 22821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:43:11.101944 22821 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:43:11.101959 22821 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:43:11.107244 22821 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37273,127.0.98.61:43741,127.0.98.60:43471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:43:11.108255 22821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:43:11.109253 22821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:43:11.112706 22740 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43471 } attempt: 2
I20260927 02:43:11.113564 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:11.113230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43471 (local address 127.0.0.1:58202)
0927 02:43:11.113398 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:11.113475 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 02:43:11.114374 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:11.114141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43471 (local address 127.0.0.1:58214)
0927 02:43:11.114233 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:11.114306 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 02:43:11.114511 22740 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43471: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43471: connect: Connection refused (error 111)
I20260927 02:43:11.115689 22815 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43471 } attempt: 2
I20260927 02:43:11.116417 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:11.116105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43471 (local address 127.0.0.1:58224)
0927 02:43:11.116280 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:11.116342 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260927 02:43:11.117159 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:11.116914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43471 (local address 127.0.0.1:58236)
0927 02:43:11.117047 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:11.117093 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260927 02:43:11.117298 22815 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43471: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43471: connect: Connection refused (error 111)
W20260927 02:43:11.118045 22830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:11.118052 22828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:11.118104 22827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:43:11.118239 22821 server_base.cc:1034] running on GCE node
I20260927 02:43:11.118958 22821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:43:11.120036 22821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:43:11.121197 22821 hybrid_clock.cc:648] HybridClock initialized: now 1790476991121179 us; error 28 us; skew 500 ppm
Sep 27 02:43:11 dist-test-slave-zv8z krb5kdc[22659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476991, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:43:11.123795 22821 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 02:43:11.124635 22821 webserver.cc:460] Webserver started at http://127.0.98.60:36823/ using document root <none> and password file <none>
I20260927 02:43:11.125157 22821 fs_manager.cc:359] Metadata directory not provided
I20260927 02:43:11.125219 22821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:43:11.125409 22821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:43:11.127466 22821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/minicluster-data/master-2/data/instance:
uuid: "29475b1da069438e9325354f6a3b1daa"
format_stamp: "Formatted at 2026-09-27 02:43:11 on dist-test-slave-zv8z"
server_key: "ddc344c068737389343a669687c6b304"
server_key_iv: "4e211918567b2e038000e86f0c7a81d8"
server_key_version: "encryptionkey@0"
I20260927 02:43:11.127990 22821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/minicluster-data/master-2/wal/instance:
uuid: "29475b1da069438e9325354f6a3b1daa"
format_stamp: "Formatted at 2026-09-27 02:43:11 on dist-test-slave-zv8z"
server_key: "ddc344c068737389343a669687c6b304"
server_key_iv: "4e211918567b2e038000e86f0c7a81d8"
server_key_version: "encryptionkey@0"
I20260927 02:43:11.131845 22821 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 02:43:11.134284 22837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:11.135560 22821 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:43:11.135671 22821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/minicluster-data/master-2/wal
uuid: "29475b1da069438e9325354f6a3b1daa"
format_stamp: "Formatted at 2026-09-27 02:43:11 on dist-test-slave-zv8z"
server_key: "ddc344c068737389343a669687c6b304"
server_key_iv: "4e211918567b2e038000e86f0c7a81d8"
server_key_version: "encryptionkey@0"
I20260927 02:43:11.135767 22821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:43:11.145278 22821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:43:11.148239 22821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:43:11.148393 22821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:43:11.154755 22821 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:43471
I20260927 02:43:11.155015 22889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:43471 every 8 connection(s)
I20260927 02:43:11.155681 22821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/minicluster-data/master-2/data/info.pb
I20260927 02:43:11.158444 22890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:43:11.162065 22890 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37273 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:43:11.163786 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 22821
I20260927 02:43:11.163890 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790476718023131-392-0/minicluster-data/master-2/wal/instance
I20260927 02:43:11.164184 392 external_mini_cluster.cc:1428] Setting key f7e96eea425959a31e104cbcadec992e
Sep 27 02:43:11 dist-test-slave-zv8z krb5kdc[22659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476991, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 02:43:11 dist-test-slave-zv8z krb5kdc[22659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 02:43:11.174680 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:11.164056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46504 (local address 127.0.98.62:37273)
0927 02:43:11.164182 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:11.164186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:11.165028 (+ 842us) server_negotiation.cc:407] Connection header received
0927 02:43:11.165984 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:11.165987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:11.166071 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:11.166139 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:11.166954 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:11.167441 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:11.168362 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:11.168569 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:11.168653 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:11.171055 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:11.171058 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:11.171061 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:11.171088 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:11.172579 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:11.173132 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:11.173135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:11.173136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:11.173207 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:11.173576 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:11.173580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:11.173582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:11.173790 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:11.173885 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:11.174403 (+ 518us) server_negotiation.cc:299] Negotiation successful
0927 02:43:11.174522 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 02:43:11.175494 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:11.164419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37273 (local address 127.0.0.1:46504)
0927 02:43:11.164878 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:11.164929 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:11.165787 (+ 858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:11.166206 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:11.166212 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:11.166568 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:43:11.166820 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:11.166832 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:11.167539 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:11.167542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:11.168251 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:11.168259 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:11.168694 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:11.168697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:11.168906 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:11.169438 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:43:11.169458 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:43:11.170917 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:43:11.172692 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:11.172700 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:11.172707 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:11.173030 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:11.173288 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:11.173292 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:11.173294 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:11.173484 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:11.173965 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:43:11.173970 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:43:11.174145 (+ 175us) client_negotiation.cc:769] Sending connection context
0927 02:43:11.174342 (+ 197us) client_negotiation.cc:240] Negotiation successful
0927 02:43:11.174658 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":31,"spinlock_wait_cycles":4736,"thread_start_us":106,"threads_started":1}
I20260927 02:43:11.175494 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:11.164800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58238 (local address 127.0.98.60:43471)
0927 02:43:11.165189 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:11.165200 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:11.165311 (+ 111us) server_negotiation.cc:407] Connection header received
0927 02:43:11.165809 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:11.165832 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:11.166017 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:11.166189 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:11.166979 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:11.167798 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:11.168599 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:11.168996 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:11.169083 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:11.171433 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:11.171438 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:11.171451 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:11.171475 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:11.172990 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:11.173443 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:11.173447 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:11.173448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:11.173527 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:11.173759 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:11.173762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:11.173763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:11.174135 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:11.174323 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:11.174554 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 02:43:11.174714 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":303,"spinlock_wait_cycles":2304,"thread_start_us":128,"threads_started":1}
I20260927 02:43:11.176630 22890 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43741 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:43:11 dist-test-slave-zv8z krb5kdc[22659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476991, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 02:43:11 dist-test-slave-zv8z krb5kdc[22659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:43:11.184758 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:11.177015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43741 (local address 127.0.0.1:42494)
0927 02:43:11.177111 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:11.177124 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:11.177185 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:11.177448 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:11.177451 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:11.177594 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:43:11.177691 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:11.177698 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:11.178793 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:11.178795 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:11.179323 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:11.179331 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:11.179795 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:11.179799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:11.179900 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:11.180342 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:43:11.180352 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:43:11.181630 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:43:11.183473 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:11.183476 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:11.183477 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:11.183653 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:11.183905 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:11.183907 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:11.183908 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:11.183968 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:11.184311 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:43:11.184314 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:43:11.184407 (+ 93us) client_negotiation.cc:769] Sending connection context
0927 02:43:11.184488 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 02:43:11.184604 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 02:43:11.184798 22820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:11.177145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42494 (local address 127.0.98.61:43741)
0927 02:43:11.177257 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:11.177261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:11.177275 (+ 14us) server_negotiation.cc:407] Connection header received
0927 02:43:11.177315 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:11.177318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:11.177363 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:11.177425 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:11.177892 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:11.178670 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:11.179470 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:11.179683 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:11.179768 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:11.181760 (+ 1992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:11.181764 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:11.181766 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:11.181795 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:11.183385 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:11.183740 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:11.183744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:11.183746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:11.183828 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:11.184041 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:11.184045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:11.184047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:11.184228 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:11.184296 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:11.184560 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 02:43:11.184651 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 02:43:11.185401 22890 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43471 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:43:11.185743 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:11.177785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46510 (local address 127.0.98.62:37273)
0927 02:43:11.177877 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:11.177880 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:11.177892 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:43:11.177967 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:11.177969 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:11.178001 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:11.178059 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:11.178697 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:11.179157 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:11.179968 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:11.180173 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:11.180246 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:11.182704 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:11.182706 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:11.182708 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:11.182729 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:11.183990 (+ 1261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:11.184374 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:11.184376 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:11.184378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:11.184440 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:11.184689 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:11.184692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:11.184694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:11.184880 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:11.185090 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:11.185392 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 02:43:11.185492 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
Sep 27 02:43:11 dist-test-slave-zv8z krb5kdc[22659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476991, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 02:43:11 dist-test-slave-zv8z krb5kdc[22659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 02:43:11.193492 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:11.185878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58240 (local address 127.0.98.60:43471)
0927 02:43:11.185973 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:11.185976 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:11.186109 (+ 133us) server_negotiation.cc:407] Connection header received
0927 02:43:11.186192 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:11.186222 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:11.186279 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:11.186344 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:11.186658 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:11.187401 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:11.188409 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:11.188754 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:11.188846 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:11.190475 (+ 1629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:11.190478 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:11.190482 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:11.190507 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:11.191838 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:11.192288 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:11.192292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:11.192294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:11.192385 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:11.192665 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:11.192668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:11.192669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:11.192887 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:11.192978 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:11.193215 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 02:43:11.193359 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":38}
I20260927 02:43:11.193497 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:11.185814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43471 (local address 127.0.0.1:58240)
0927 02:43:11.186042 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:11.186055 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:11.186129 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:11.186353 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:11.186356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:11.186510 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:43:11.186567 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:11.186572 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:11.187525 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:11.187528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:11.188297 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:11.188306 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:11.188861 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:11.188863 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:11.188919 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:11.189298 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:43:11.189311 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:43:11.190352 (+ 1041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:43:11.191934 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:11.191938 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:11.191939 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:11.192161 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:11.192469 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:11.192471 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:11.192472 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:11.192576 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:11.192976 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:43:11.192979 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:43:11.193072 (+ 93us) client_negotiation.cc:769] Sending connection context
0927 02:43:11.193180 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 02:43:11.193306 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177}
I20260927 02:43:11.195233 22820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:11.186731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42510 (local address 127.0.98.61:43741)
0927 02:43:11.186824 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:11.186827 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:11.186837 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:43:11.186931 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:11.186933 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:11.186963 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:11.187020 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:11.187650 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:11.188368 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:11.189238 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:11.189464 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:11.189550 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:11.192008 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:11.192020 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:11.192022 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:11.192046 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:11.193660 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:11.194094 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:11.194096 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:11.194097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:11.194171 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:11.194478 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:11.194481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:11.194483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:11.194665 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:11.194886 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:11.194999 (+ 113us) server_negotiation.cc:299] Negotiation successful
0927 02:43:11.195096 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 02:43:11.197093 22890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29475b1da069438e9325354f6a3b1daa: Bootstrap starting.
I20260927 02:43:11.198364 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:43:11.199644 22890 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29475b1da069438e9325354f6a3b1daa: Neither blocks nor log segments found. Creating new log.
I20260927 02:43:11.200356 22890 log.cc:824] T 00000000000000000000000000000000 P 29475b1da069438e9325354f6a3b1daa: Log is configured to *not* fsync() on all Append() calls
I20260927 02:43:11.199512 22714 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "978d68c4839d4229ae8256d05b55e015"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46510
I20260927 02:43:11.202414 22890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29475b1da069438e9325354f6a3b1daa: No bootstrap required, opened a new log
I20260927 02:43:11.206352 22890 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29475b1da069438e9325354f6a3b1daa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "978d68c4839d4229ae8256d05b55e015" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37273 } } peers { permanent_uuid: "46989b41783443839c9fabb12ea57d73" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43741 } } peers { permanent_uuid: "29475b1da069438e9325354f6a3b1daa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43471 } }
I20260927 02:43:11.206597 22890 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29475b1da069438e9325354f6a3b1daa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29475b1da069438e9325354f6a3b1daa, State: Initialized, Role: FOLLOWER
I20260927 02:43:11.206933 22890 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29475b1da069438e9325354f6a3b1daa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "978d68c4839d4229ae8256d05b55e015" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37273 } } peers { permanent_uuid: "46989b41783443839c9fabb12ea57d73" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43741 } } peers { permanent_uuid: "29475b1da069438e9325354f6a3b1daa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43471 } }
I20260927 02:43:11.207383 22896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29475b1da069438e9325354f6a3b1daa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "978d68c4839d4229ae8256d05b55e015" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37273 } } peers { permanent_uuid: "46989b41783443839c9fabb12ea57d73" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43741 } } peers { permanent_uuid: "29475b1da069438e9325354f6a3b1daa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43471 } } }
I20260927 02:43:11.207501 22896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29475b1da069438e9325354f6a3b1daa [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:43:11.208271 22890 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29475b1da069438e9325354f6a3b1daa [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:43:11.208433 22815 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43471 } attempt: 3
W20260927 02:43:11.213114 22907 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 29475b1da069438e9325354f6a3b1daa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:43:11.213202 22907 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 27 02:43:11 dist-test-slave-zv8z krb5kdc[22659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 02:43:11.213937 22740 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43471 } attempt: 3
I20260927 02:43:11.212062 22821 master_runner.cc:186] Error getting master registration for 127.0.98.62:37273: OK, instance_id { permanent_uuid: "978d68c4839d4229ae8256d05b55e015" instance_seqno: 1790476990751051 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 02:43:11.215646 22821 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 02:43:11.217913 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:11.208863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43471 (local address 127.0.0.1:58244)
0927 02:43:11.208977 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:11.208992 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:11.209075 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:11.209367 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:11.209371 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:11.209570 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:43:11.209683 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:11.209691 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:11.210799 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:11.210803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:11.211520 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:11.211531 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:11.212080 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:11.212084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:11.212171 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:11.212687 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:43:11.212703 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:43:11.214436 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:43:11.216303 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:11.216315 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:11.216317 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:11.216553 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:11.216847 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:11.216850 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:11.216852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:11.216945 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:11.217420 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:43:11.217424 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:43:11.217540 (+ 116us) client_negotiation.cc:769] Sending connection context
0927 02:43:11.217641 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 02:43:11.217779 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 02:43:11.218355 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:11.208922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58244 (local address 127.0.98.60:43471)
0927 02:43:11.209074 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:11.209079 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:11.209098 (+ 19us) server_negotiation.cc:407] Connection header received
0927 02:43:11.209152 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:11.209156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:11.209201 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:11.209802 (+ 601us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:11.209893 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:11.210654 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:11.211652 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:11.211958 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:11.212408 (+ 450us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:11.214580 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:11.214583 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:11.214586 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:11.214610 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:11.216202 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:11.216644 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:11.216647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:11.216649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:11.216735 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:11.217066 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:11.217070 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:11.217072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:11.217307 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:11.217403 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:11.218096 (+ 693us) server_negotiation.cc:299] Negotiation successful
0927 02:43:11.218193 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Sep 27 02:43:11 dist-test-slave-zv8z krb5kdc[22659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 02:43:11.221666 22815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46989b41783443839c9fabb12ea57d73: Bootstrap starting.
I20260927 02:43:11.224493 22815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46989b41783443839c9fabb12ea57d73: Neither blocks nor log segments found. Creating new log.
I20260927 02:43:11.224723 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:11.214435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43471 (local address 127.0.0.1:58246)
0927 02:43:11.214605 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:11.214620 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:11.214696 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:11.216435 (+ 1739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:11.216438 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:11.216606 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:43:11.216716 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:11.216722 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:11.217780 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:11.217784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:11.218668 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:11.218679 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:11.219243 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:11.219247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:11.219342 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:11.219773 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:43:11.219787 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:43:11.221050 (+ 1263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:43:11.223181 (+ 2131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:11.223184 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:11.223186 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:11.223381 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:11.223678 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:11.223681 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:11.223683 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:11.223778 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:11.224190 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:43:11.224194 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:43:11.224283 (+ 89us) client_negotiation.cc:769] Sending connection context
0927 02:43:11.224371 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 02:43:11.224507 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260927 02:43:11.224754 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:11.215815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58246 (local address 127.0.98.60:43471)
0927 02:43:11.216149 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:11.216157 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:11.216194 (+ 37us) server_negotiation.cc:407] Connection header received
0927 02:43:11.216251 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:11.216254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:11.216314 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:11.216400 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:11.216821 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:11.217645 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:11.218782 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:11.219113 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:11.219215 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:11.221183 (+ 1968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:11.221187 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:11.221189 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:11.221228 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:11.223085 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:11.223467 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:11.223470 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:11.223472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:11.223550 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:11.223872 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:11.223875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:11.223877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:11.224095 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:11.224176 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:11.224433 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 02:43:11.224542 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":106,"threads_started":1}
I20260927 02:43:11.225503 22815 log.cc:824] T 00000000000000000000000000000000 P 46989b41783443839c9fabb12ea57d73: Log is configured to *not* fsync() on all Append() calls
I20260927 02:43:11.227598 22740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 978d68c4839d4229ae8256d05b55e015: Bootstrap starting.
I20260927 02:43:11.228469 22815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46989b41783443839c9fabb12ea57d73: No bootstrap required, opened a new log
I20260927 02:43:11.229914 22740 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 978d68c4839d4229ae8256d05b55e015: Neither blocks nor log segments found. Creating new log.
I20260927 02:43:11.230665 22740 log.cc:824] T 00000000000000000000000000000000 P 978d68c4839d4229ae8256d05b55e015: Log is configured to *not* fsync() on all Append() calls
I20260927 02:43:11.232770 22740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 978d68c4839d4229ae8256d05b55e015: No bootstrap required, opened a new log
I20260927 02:43:11.233268 22815 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46989b41783443839c9fabb12ea57d73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "978d68c4839d4229ae8256d05b55e015" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37273 } } peers { permanent_uuid: "46989b41783443839c9fabb12ea57d73" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43741 } } peers { permanent_uuid: "29475b1da069438e9325354f6a3b1daa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43471 } }
I20260927 02:43:11.233484 22815 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46989b41783443839c9fabb12ea57d73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46989b41783443839c9fabb12ea57d73, State: Initialized, Role: FOLLOWER
I20260927 02:43:11.233803 22815 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46989b41783443839c9fabb12ea57d73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "978d68c4839d4229ae8256d05b55e015" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37273 } } peers { permanent_uuid: "46989b41783443839c9fabb12ea57d73" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43741 } } peers { permanent_uuid: "29475b1da069438e9325354f6a3b1daa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43471 } }
I20260927 02:43:11.234428 22912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46989b41783443839c9fabb12ea57d73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "978d68c4839d4229ae8256d05b55e015" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37273 } } peers { permanent_uuid: "46989b41783443839c9fabb12ea57d73" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43741 } } peers { permanent_uuid: "29475b1da069438e9325354f6a3b1daa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43471 } } }
I20260927 02:43:11.234714 22912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46989b41783443839c9fabb12ea57d73 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:43:11.235134 22815 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46989b41783443839c9fabb12ea57d73 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:43:11.235677 22740 raft_consensus.cc:348] T 00000000000000000000000000000000 P 978d68c4839d4229ae8256d05b55e015 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "978d68c4839d4229ae8256d05b55e015" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37273 } } peers { permanent_uuid: "46989b41783443839c9fabb12ea57d73" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43741 } } peers { permanent_uuid: "29475b1da069438e9325354f6a3b1daa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43471 } }
I20260927 02:43:11.235888 22740 raft_consensus.cc:729] T 00000000000000000000000000000000 P 978d68c4839d4229ae8256d05b55e015 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 978d68c4839d4229ae8256d05b55e015, State: Initialized, Role: FOLLOWER
I20260927 02:43:11.236212 22740 consensus_queue.cc:260] T 00000000000000000000000000000000 P 978d68c4839d4229ae8256d05b55e015 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "978d68c4839d4229ae8256d05b55e015" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37273 } } peers { permanent_uuid: "46989b41783443839c9fabb12ea57d73" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43741 } } peers { permanent_uuid: "29475b1da069438e9325354f6a3b1daa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43471 } }
I20260927 02:43:11.236764 22918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 978d68c4839d4229ae8256d05b55e015 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "978d68c4839d4229ae8256d05b55e015" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37273 } } peers { permanent_uuid: "46989b41783443839c9fabb12ea57d73" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43741 } } peers { permanent_uuid: "29475b1da069438e9325354f6a3b1daa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43471 } } }
I20260927 02:43:11.237021 22918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 978d68c4839d4229ae8256d05b55e015 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:43:11.237573 22740 sys_catalog.cc:564] T 00000000000000000000000000000000 P 978d68c4839d4229ae8256d05b55e015 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:43:11.237624 22714 raft_consensus.cc:482] T 00000000000000000000000000000000 P 978d68c4839d4229ae8256d05b55e015 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 02:43:11.237723 22714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 978d68c4839d4229ae8256d05b55e015 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:43:11.239073 22714 raft_consensus.cc:504] T 00000000000000000000000000000000 P 978d68c4839d4229ae8256d05b55e015 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "978d68c4839d4229ae8256d05b55e015" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37273 } } peers { permanent_uuid: "46989b41783443839c9fabb12ea57d73" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43741 } } peers { permanent_uuid: "29475b1da069438e9325354f6a3b1daa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43471 } }
I20260927 02:43:11.240064 22714 leader_election.cc:288] T 00000000000000000000000000000000 P 978d68c4839d4229ae8256d05b55e015 [CANDIDATE]: Term 1 election: Requested vote from peers 46989b41783443839c9fabb12ea57d73 (127.0.98.61:43741), 29475b1da069438e9325354f6a3b1daa (127.0.98.60:43471)
I20260927 02:43:11.240816 22789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "978d68c4839d4229ae8256d05b55e015" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "46989b41783443839c9fabb12ea57d73"
I20260927 02:43:11.241060 22789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46989b41783443839c9fabb12ea57d73 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:43:11.241015 22864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "978d68c4839d4229ae8256d05b55e015" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "29475b1da069438e9325354f6a3b1daa"
W20260927 02:43:11.241118 22924 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 46989b41783443839c9fabb12ea57d73: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:43:11.241304 22924 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:43:11.241291 22864 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29475b1da069438e9325354f6a3b1daa [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:43:11.238703 22747 master_runner.cc:186] Error getting master registration for 127.0.98.62:37273: OK, instance_id { permanent_uuid: "978d68c4839d4229ae8256d05b55e015" instance_seqno: 1790476990751051 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 02:43:11.242487 22864 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 29475b1da069438e9325354f6a3b1daa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 978d68c4839d4229ae8256d05b55e015 in term 1.
I20260927 02:43:11.242998 22747 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 02:43:11.243278 22688 leader_election.cc:302] T 00000000000000000000000000000000 P 978d68c4839d4229ae8256d05b55e015 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29475b1da069438e9325354f6a3b1daa, 978d68c4839d4229ae8256d05b55e015; no voters:
I20260927 02:43:11.243538 22918 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 978d68c4839d4229ae8256d05b55e015 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:43:11.243743 22918 raft_consensus.cc:686] T 00000000000000000000000000000000 P 978d68c4839d4229ae8256d05b55e015 [term 1 LEADER]: Becoming Leader. State: Replica: 978d68c4839d4229ae8256d05b55e015, State: Running, Role: LEADER
I20260927 02:43:11.244060 22918 consensus_queue.cc:237] T 00000000000000000000000000000000 P 978d68c4839d4229ae8256d05b55e015 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "978d68c4839d4229ae8256d05b55e015" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37273 } } peers { permanent_uuid: "46989b41783443839c9fabb12ea57d73" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43741 } } peers { permanent_uuid: "29475b1da069438e9325354f6a3b1daa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43471 } }
I20260927 02:43:11.242395 22789 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 46989b41783443839c9fabb12ea57d73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 978d68c4839d4229ae8256d05b55e015 in term 1.
I20260927 02:43:11.245685 22918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 978d68c4839d4229ae8256d05b55e015 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 978d68c4839d4229ae8256d05b55e015. Latest consensus state: current_term: 1 leader_uuid: "978d68c4839d4229ae8256d05b55e015" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "978d68c4839d4229ae8256d05b55e015" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37273 } } peers { permanent_uuid: "46989b41783443839c9fabb12ea57d73" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43741 } } peers { permanent_uuid: "29475b1da069438e9325354f6a3b1daa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43471 } } }
I20260927 02:43:11.245775 22918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 978d68c4839d4229ae8256d05b55e015 [sys.catalog]: This master's current role is: LEADER
W20260927 02:43:11.247736 22693 tablet.cc:2369] T 00000000000000000000000000000000 P 978d68c4839d4229ae8256d05b55e015: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:43:11.250065 22672 master_runner.cc:263] No leader master found from master 978d68c4839d4229ae8256d05b55e015
I20260927 02:43:11.250121 22672 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 02:43:11.255846 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:11.245519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46520 (local address 127.0.98.62:37273)
0927 02:43:11.246087 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:11.246092 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:11.246940 (+ 848us) server_negotiation.cc:407] Connection header received
0927 02:43:11.247127 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:11.247130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:11.247176 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:11.247223 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:11.248203 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:11.249049 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:11.250214 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:11.250557 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:11.250622 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:11.252426 (+ 1804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:11.252430 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:11.252432 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:11.252461 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:11.253978 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:11.254470 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:11.254474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:11.254476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:11.254558 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:11.254907 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:11.254911 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:11.254913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:11.255123 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:11.255234 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:11.255582 (+ 348us) server_negotiation.cc:299] Negotiation successful
0927 02:43:11.255694 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501}
I20260927 02:43:11.256285 22945 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:43:11.257629 22820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:11.245527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42526 (local address 127.0.98.61:43741)
0927 02:43:11.245631 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:11.245635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:11.248531 (+ 2896us) server_negotiation.cc:407] Connection header received
0927 02:43:11.248698 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:11.248702 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:11.248745 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:11.248789 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:11.249682 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:11.250430 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:11.251449 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:11.251818 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:11.251899 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:11.253687 (+ 1788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:11.253691 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:11.253694 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:11.253720 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:11.255281 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:11.255707 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:11.255710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:11.255712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:11.255788 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:11.256075 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:11.256079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:11.256081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:11.256278 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:11.256385 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:11.257351 (+ 966us) server_negotiation.cc:299] Negotiation successful
0927 02:43:11.257477 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 02:43:11.259719 22945 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:43:11.264566 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:11.246363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58262 (local address 127.0.98.60:43471)
0927 02:43:11.246475 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:11.246479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:11.256357 (+ 9878us) server_negotiation.cc:407] Connection header received
0927 02:43:11.256538 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:11.256541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:11.256591 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:11.256640 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:11.257374 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:11.258480 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:11.259473 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:11.259847 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:11.259941 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:11.261216 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:11.261219 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:11.261222 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:11.261249 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:11.262764 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:11.263228 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:11.263232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:11.263234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:11.263317 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:11.263602 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:11.263605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:11.263606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:11.263812 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:11.263925 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:11.264291 (+ 366us) server_negotiation.cc:299] Negotiation successful
0927 02:43:11.264416 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 02:43:11.265659 22864 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 29475b1da069438e9325354f6a3b1daa [term 1 FOLLOWER]: Refusing update from remote peer 978d68c4839d4229ae8256d05b55e015: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:43:11.266005 22789 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 46989b41783443839c9fabb12ea57d73 [term 1 FOLLOWER]: Refusing update from remote peer 978d68c4839d4229ae8256d05b55e015: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:43:11.266521 22938 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 978d68c4839d4229ae8256d05b55e015 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29475b1da069438e9325354f6a3b1daa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:43:11.266759 22918 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 978d68c4839d4229ae8256d05b55e015 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46989b41783443839c9fabb12ea57d73" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:43:11.270234 22896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29475b1da069438e9325354f6a3b1daa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 978d68c4839d4229ae8256d05b55e015. Latest consensus state: current_term: 1 leader_uuid: "978d68c4839d4229ae8256d05b55e015" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "978d68c4839d4229ae8256d05b55e015" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37273 } } peers { permanent_uuid: "46989b41783443839c9fabb12ea57d73" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43741 } } peers { permanent_uuid: "29475b1da069438e9325354f6a3b1daa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43471 } } }
I20260927 02:43:11.270367 22896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29475b1da069438e9325354f6a3b1daa [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:43:11.270740 22912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46989b41783443839c9fabb12ea57d73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 978d68c4839d4229ae8256d05b55e015. Latest consensus state: current_term: 1 leader_uuid: "978d68c4839d4229ae8256d05b55e015" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "978d68c4839d4229ae8256d05b55e015" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37273 } } peers { permanent_uuid: "46989b41783443839c9fabb12ea57d73" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43741 } } peers { permanent_uuid: "29475b1da069438e9325354f6a3b1daa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43471 } } }
I20260927 02:43:11.270825 22912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46989b41783443839c9fabb12ea57d73 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:43:11.272909 22918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 978d68c4839d4229ae8256d05b55e015 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "978d68c4839d4229ae8256d05b55e015" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "978d68c4839d4229ae8256d05b55e015" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37273 } } peers { permanent_uuid: "46989b41783443839c9fabb12ea57d73" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43741 } } peers { permanent_uuid: "29475b1da069438e9325354f6a3b1daa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43471 } } }
I20260927 02:43:11.273029 22918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 978d68c4839d4229ae8256d05b55e015 [sys.catalog]: This master's current role is: LEADER
I20260927 02:43:11.273487 22912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46989b41783443839c9fabb12ea57d73 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "978d68c4839d4229ae8256d05b55e015" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "978d68c4839d4229ae8256d05b55e015" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37273 } } peers { permanent_uuid: "46989b41783443839c9fabb12ea57d73" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43741 } } peers { permanent_uuid: "29475b1da069438e9325354f6a3b1daa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43471 } } }
I20260927 02:43:11.273587 22912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46989b41783443839c9fabb12ea57d73 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:43:11.276073 22949 mvcc.cc:205] Tried to move back new op lower bound from 7333793756214931456 to 7333793756138786816. Current Snapshot: MvccSnapshot[applied={T|T < 7333793756214931456}]
I20260927 02:43:11.280313 22918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 978d68c4839d4229ae8256d05b55e015 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "978d68c4839d4229ae8256d05b55e015" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "978d68c4839d4229ae8256d05b55e015" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37273 } } peers { permanent_uuid: "46989b41783443839c9fabb12ea57d73" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43741 } } peers { permanent_uuid: "29475b1da069438e9325354f6a3b1daa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43471 } } }
I20260927 02:43:11.280435 22918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 978d68c4839d4229ae8256d05b55e015 [sys.catalog]: This master's current role is: LEADER
I20260927 02:43:11.281276 22896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29475b1da069438e9325354f6a3b1daa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "978d68c4839d4229ae8256d05b55e015" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "978d68c4839d4229ae8256d05b55e015" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37273 } } peers { permanent_uuid: "46989b41783443839c9fabb12ea57d73" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43741 } } peers { permanent_uuid: "29475b1da069438e9325354f6a3b1daa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43471 } } }
I20260927 02:43:11.281387 22896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29475b1da069438e9325354f6a3b1daa [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:43:11.281808 22950 mvcc.cc:205] Tried to move back new op lower bound from 7333793756214931456 to 7333793756138786816. Current Snapshot: MvccSnapshot[applied={T|T < 7333793756214931456}]
I20260927 02:43:11.283530 22945 catalog_manager.cc:1347] Generated new cluster ID: 6b39e62a604c41d49e5626bf26f573de
I20260927 02:43:11.283630 22945 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:43:11.305496 22945 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:43:11.307088 22945 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:43:11.316807 22945 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 978d68c4839d4229ae8256d05b55e015: Generated new TSK 0
I20260927 02:43:11.317598 22945 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 02:43:11.361842 22743 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:43:11.358897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46522 (local address 127.0.98.62:37273)
0927 02:43:11.359038 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:11.359043 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:11.360783 (+ 1740us) server_negotiation.cc:407] Connection header received
0927 02:43:11.360986 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:11.360990 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:11.361040 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:11.361092 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:11.361702 (+ 610us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46522: BlockingRecv error: recv got EOF from 127.0.0.1:46522 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260927 02:43:11.363197 22820 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:43:11.359629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42528 (local address 127.0.98.61:43741)
0927 02:43:11.359738 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:11.359742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:11.362332 (+ 2590us) server_negotiation.cc:407] Connection header received
0927 02:43:11.362500 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:11.362503 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:11.362551 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:11.362600 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:11.363083 (+ 483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42528: BlockingRecv error: recv got EOF from 127.0.0.1:42528 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260927 02:43:11.364615 22909 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:43:11.360285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58276 (local address 127.0.98.60:43471)
0927 02:43:11.360389 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:11.360393 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:11.363556 (+ 3163us) server_negotiation.cc:407] Connection header received
0927 02:43:11.363899 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:11.363902 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:11.363962 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:11.364012 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:11.364506 (+ 494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58276: BlockingRecv error: recv got EOF from 127.0.0.1:58276 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260927 02:43:11.368788 22820 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:43:11.367385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42544 (local address 127.0.98.61:43741)
0927 02:43:11.367480 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:11.367484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:11.367922 (+ 438us) server_negotiation.cc:407] Connection header received
0927 02:43:11.368135 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:11.368138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:11.368208 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:11.368255 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:11.368695 (+ 440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42544: BlockingRecv error: recv got EOF from 127.0.0.1:42544 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260927 02:43:11.373451 22820 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:43:11.372118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42560 (local address 127.0.98.61:43741)
0927 02:43:11.372208 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:11.372212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:11.372655 (+ 443us) server_negotiation.cc:407] Connection header received
0927 02:43:11.372818 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:11.372821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:11.372864 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:11.372909 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:11.373358 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42560: BlockingRecv error: recv got EOF from 127.0.0.1:42560 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260927 02:43:11.378925 22820 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:43:11.377430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42564 (local address 127.0.98.61:43741)
0927 02:43:11.377523 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:11.377526 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:11.377997 (+ 471us) server_negotiation.cc:407] Connection header received
0927 02:43:11.378225 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:11.378228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:11.378330 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:11.378379 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:11.378833 (+ 454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42564: BlockingRecv error: recv got EOF from 127.0.0.1:42564 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260927 02:43:11.386374 22820 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:43:11.384893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42574 (local address 127.0.98.61:43741)
0927 02:43:11.385026 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:11.385029 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:11.385493 (+ 464us) server_negotiation.cc:407] Connection header received
0927 02:43:11.385659 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:11.385663 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:11.385708 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:11.385754 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:11.386219 (+ 465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42574: BlockingRecv error: recv got EOF from 127.0.0.1:42574 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260927 02:43:11.398222 22820 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:43:11.396600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42578 (local address 127.0.98.61:43741)
0927 02:43:11.396709 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:11.396713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:11.397256 (+ 543us) server_negotiation.cc:407] Connection header received
0927 02:43:11.397449 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:11.397452 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:11.397503 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:11.397555 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:11.398112 (+ 557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42578: BlockingRecv error: recv got EOF from 127.0.0.1:42578 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260927 02:43:11.418284 22820 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:43:11.416922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42592 (local address 127.0.98.61:43741)
0927 02:43:11.417049 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:11.417053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:11.417317 (+ 264us) server_negotiation.cc:407] Connection header received
0927 02:43:11.417464 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:11.417467 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:11.417529 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:11.417637 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:11.418137 (+ 500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42592: BlockingRecv error: recv got EOF from 127.0.0.1:42592 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 02:43:11.453384 22820 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:43:11.452245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42602 (local address 127.0.98.61:43741)
0927 02:43:11.452428 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:11.452433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:11.452547 (+ 114us) server_negotiation.cc:407] Connection header received
0927 02:43:11.452676 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:11.452680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:11.452737 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:11.452837 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:11.453270 (+ 433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42602: BlockingRecv error: recv got EOF from 127.0.0.1:42602 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260927 02:43:11.520527 22820 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:43:11.519465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42606 (local address 127.0.98.61:43741)
0927 02:43:11.519612 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:11.519616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:11.519716 (+ 100us) server_negotiation.cc:407] Connection header received
0927 02:43:11.519868 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:11.519870 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:11.519925 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:11.519992 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:11.520430 (+ 438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42606: BlockingRecv error: recv got EOF from 127.0.0.1:42606 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260927 02:43:11.651567 22820 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:43:11.650564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42620 (local address 127.0.98.61:43741)
0927 02:43:11.650706 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:11.650710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:11.650823 (+ 113us) server_negotiation.cc:407] Connection header received
0927 02:43:11.650944 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:11.650947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:11.650991 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:11.651068 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:11.651471 (+ 403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42620: BlockingRecv error: recv got EOF from 127.0.0.1:42620 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 02:43:11.910751 22820 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:43:11.909629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42624 (local address 127.0.98.61:43741)
0927 02:43:11.909773 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:11.909778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:11.909875 (+ 97us) server_negotiation.cc:407] Connection header received
0927 02:43:11.910025 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:11.910028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:11.910081 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:11.910150 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:11.910649 (+ 499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42624: BlockingRecv error: recv got EOF from 127.0.0.1:42624 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 02:43:12.214491 22907 catalog_manager.cc:1259] Loaded cluster ID: 6b39e62a604c41d49e5626bf26f573de
I20260927 02:43:12.214582 22907 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 29475b1da069438e9325354f6a3b1daa: loading cluster ID for follower catalog manager: success
I20260927 02:43:12.216185 22907 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 29475b1da069438e9325354f6a3b1daa: acquiring CA information for follower catalog manager: success
I20260927 02:43:12.217252 22907 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 29475b1da069438e9325354f6a3b1daa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 02:43:12.242640 22924 catalog_manager.cc:1259] Loaded cluster ID: 6b39e62a604c41d49e5626bf26f573de
I20260927 02:43:12.242713 22924 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 46989b41783443839c9fabb12ea57d73: loading cluster ID for follower catalog manager: success
I20260927 02:43:12.244297 22924 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 46989b41783443839c9fabb12ea57d73: acquiring CA information for follower catalog manager: success
I20260927 02:43:12.245389 22924 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 46989b41783443839c9fabb12ea57d73: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 02:43:12.428678 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:12.424891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42630 (local address 127.0.98.61:43741)
0927 02:43:12.425153 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:12.425157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:12.425174 (+ 17us) server_negotiation.cc:407] Connection header received
0927 02:43:12.425298 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:12.425301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:12.425420 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:12.425517 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:43:12.425798 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:12.426339 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:12.427196 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:12.427423 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:12.427510 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:12.427811 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:43:12.428238 (+ 427us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:43:12.428347 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:12.428415 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 02:43:12.428507 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":69,"threads_started":1}
I20260927 02:43:12.435529 23028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:12.431667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58282 (local address 127.0.98.60:43471)
0927 02:43:12.431882 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:12.431885 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:12.432036 (+ 151us) server_negotiation.cc:407] Connection header received
0927 02:43:12.432172 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:12.432175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:12.432323 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:12.432418 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:43:12.432625 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:12.433167 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:12.433957 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:12.434213 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:12.434335 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:12.434644 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:43:12.435139 (+ 495us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:43:12.435232 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:12.435334 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 02:43:12.435380 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":77,"threads_started":1}
I20260927 02:43:12.442270 23028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:12.438720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58284 (local address 127.0.98.60:43471)
0927 02:43:12.438848 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:12.438852 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:12.438871 (+ 19us) server_negotiation.cc:407] Connection header received
0927 02:43:12.438909 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:12.438912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:12.438998 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:12.439070 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:43:12.439390 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:12.440101 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:12.441260 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:12.441503 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:12.441596 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:12.441840 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:43:12.441905 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:43:12.441960 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:12.442069 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 02:43:12.442117 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 02:43:12.443143 23035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:12.438325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46538 (local address 127.0.98.62:37273)
0927 02:43:12.438750 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:12.438755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:12.438802 (+ 47us) server_negotiation.cc:407] Connection header received
0927 02:43:12.438900 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:12.438902 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:12.439104 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:12.439225 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:43:12.439483 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:12.440314 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:12.441434 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:12.441755 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:12.441839 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:12.442222 (+ 383us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:43:12.442577 (+ 355us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:43:12.442669 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:12.442924 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 02:43:12.443004 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":70,"threads_started":1}
I20260927 02:43:12.443305 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:12.438611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42642 (local address 127.0.98.61:43741)
0927 02:43:12.438854 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:12.438858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:12.439214 (+ 356us) server_negotiation.cc:407] Connection header received
0927 02:43:12.439344 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:12.439347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:12.439395 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:12.439440 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:43:12.439837 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:12.440545 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:12.441848 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:12.442155 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:12.442219 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:12.442786 (+ 567us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:43:12.442861 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:43:12.442927 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:12.443061 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 02:43:12.443128 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163}
I20260927 02:43:12.445851 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 22672
I20260927 02:43:12.451980 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 22747
I20260927 02:43:12.456605 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 22821
2026-09-27T02:43:12Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2949 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:43:12 dist-test-slave-zv8z krb5kdc[23039](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:43:12 dist-test-slave-zv8z krb5kdc[23039](info): set up 2 sockets
Sep 27 02:43:12 dist-test-slave-zv8z krb5kdc[23039](info): commencing operation
krb5kdc: starting...
W20260927 02:43:13.410406 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/krb5kdc/test-user.keytab.
Sep 27 02:43:13 dist-test-slave-zv8z krb5kdc[23039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:43:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:43:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:43:13.522814 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36831
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36831,127.0.98.61:42719,127.0.98.60:39335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:43:13.627368 23052 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 02:43:13.627614 23052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:43:13.627635 23052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:43:13.627668 23052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:43:13.631492 23052 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:43:13.631538 23052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:43:13.631556 23052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:43:13.631570 23052 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:43:13.631582 23052 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:43:13.636585 23052 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36831,127.0.98.61:42719,127.0.98.60:39335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:43:13.637514 23052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:43:13.638448 23052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:43:13.646431 23058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:13.646497 23060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:43:13.646486 23052 server_base.cc:1034] running on GCE node
W20260927 02:43:13.646441 23057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:43:13.647141 23052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:43:13.648164 23052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:43:13.649341 23052 hybrid_clock.cc:648] HybridClock initialized: now 1790476993649315 us; error 36 us; skew 500 ppm
Sep 27 02:43:13 dist-test-slave-zv8z krb5kdc[23039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:43:13.657855 23052 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:43:13.658977 23052 webserver.cc:460] Webserver started at http://127.0.98.62:42789/ using document root <none> and password file <none>
I20260927 02:43:13.659673 23052 fs_manager.cc:359] Metadata directory not provided
I20260927 02:43:13.659763 23052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:43:13.659996 23052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:43:13.662026 23052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/minicluster-data/master-0/data/instance:
uuid: "936623e894d241ec8ce9fab364baddd8"
format_stamp: "Formatted at 2026-09-27 02:43:13 on dist-test-slave-zv8z"
server_key: "fef29be4e7dbf81e8da76c487f3fce38"
server_key_iv: "6edfdd24435bd6c99cfc96a8171c9247"
server_key_version: "encryptionkey@0"
I20260927 02:43:13.662576 23052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/minicluster-data/master-0/wal/instance:
uuid: "936623e894d241ec8ce9fab364baddd8"
format_stamp: "Formatted at 2026-09-27 02:43:13 on dist-test-slave-zv8z"
server_key: "fef29be4e7dbf81e8da76c487f3fce38"
server_key_iv: "6edfdd24435bd6c99cfc96a8171c9247"
server_key_version: "encryptionkey@0"
I20260927 02:43:13.666051 23052 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 02:43:13.668514 23067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:13.669672 23052 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:43:13.669790 23052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/minicluster-data/master-0/wal
uuid: "936623e894d241ec8ce9fab364baddd8"
format_stamp: "Formatted at 2026-09-27 02:43:13 on dist-test-slave-zv8z"
server_key: "fef29be4e7dbf81e8da76c487f3fce38"
server_key_iv: "6edfdd24435bd6c99cfc96a8171c9247"
server_key_version: "encryptionkey@0"
I20260927 02:43:13.669880 23052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:43:13.695115 23052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:43:13.698094 23052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:43:13.698287 23052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:43:13.704566 23052 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36831
I20260927 02:43:13.704839 23119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36831 every 8 connection(s)
I20260927 02:43:13.705469 23052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:43:13.708196 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 23052
I20260927 02:43:13.708237 23120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:43:13.708308 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/minicluster-data/master-0/wal/instance
I20260927 02:43:13.708576 392 external_mini_cluster.cc:1428] Setting key d4d8b1cecdf1d234a78d46625515e412
I20260927 02:43:13.711918 23120 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36831 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 27 02:43:13 dist-test-slave-zv8z krb5kdc[23039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:43:13.727632 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:13.714063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36831 (local address 127.0.0.1:54346)
0927 02:43:13.714680 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:13.714725 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:13.715560 (+ 835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:13.716238 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:13.716243 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:13.716594 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:43:13.716849 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:13.716861 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:13.718128 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:13.718131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:13.719228 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:13.719240 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:13.719866 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:13.719870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:13.719972 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:13.720525 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:43:13.720541 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:43:13.722126 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:43:13.724348 (+ 2222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:13.724356 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:13.724365 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:13.724652 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:13.725000 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:13.725003 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:13.725005 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:13.725137 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:13.725841 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:43:13.725848 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:43:13.726005 (+ 157us) client_negotiation.cc:769] Sending connection context
0927 02:43:13.726191 (+ 186us) client_negotiation.cc:240] Negotiation successful
0927 02:43:13.726702 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":116,"spinlock_wait_cycles":72448,"thread_start_us":176,"threads_started":1}
I20260927 02:43:13.727630 23123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:13.714086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54346 (local address 127.0.98.62:36831)
0927 02:43:13.714535 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:13.714554 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:13.714838 (+ 284us) server_negotiation.cc:407] Connection header received
0927 02:43:13.715828 (+ 990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:13.715832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:13.716019 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:13.716194 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:13.716969 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:13.718000 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:13.719360 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:13.719727 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:13.719867 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:13.722328 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:13.722335 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:13.722349 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:13.722375 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:13.724242 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:13.724771 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:13.724777 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:13.724778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:13.724908 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:13.725229 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:13.725232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:13.725234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:13.725716 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:13.725846 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:13.726295 (+ 449us) server_negotiation.cc:299] Negotiation successful
0927 02:43:13.726616 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":312,"thread_start_us":155,"threads_started":1}
I20260927 02:43:13.730666 23120 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42719 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:43:13.731325 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:13.731085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42719 (local address 127.0.0.1:50754)
0927 02:43:13.731190 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:13.731258 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 02:43:13.731505 23071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42719: connect: Connection refused (error 111)
I20260927 02:43:13.732239 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:13.732044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42719 (local address 127.0.0.1:50764)
0927 02:43:13.732128 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:13.732176 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 02:43:13.732372 23120 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42719: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42719: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 02:43:13.734853 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42719
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36831,127.0.98.61:42719,127.0.98.60:39335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:43:13.781611 23120 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42719 } attempt: 1
I20260927 02:43:13.782522 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:13.782176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42719 (local address 127.0.0.1:50774)
0927 02:43:13.782366 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:13.782430 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 02:43:13.783198 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:13.782986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42719 (local address 127.0.0.1:50784)
0927 02:43:13.783090 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:13.783135 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 02:43:13.783344 23120 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42719: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42719: connect: Connection refused (error 111)
W20260927 02:43:13.839353 23127 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 02:43:13.839632 23127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:43:13.839663 23127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:43:13.839722 23127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:43:13.843689 23127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:43:13.843745 23127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:43:13.843772 23127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:43:13.843792 23127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:43:13.843813 23127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:43:13.848932 23127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36831,127.0.98.61:42719,127.0.98.60:39335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:43:13.849941 23127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:43:13.850947 23127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:43:13.851491 23120 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42719 } attempt: 2
I20260927 02:43:13.852278 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:13.851955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42719 (local address 127.0.0.1:50790)
0927 02:43:13.852136 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:13.852202 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 02:43:13.852938 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:13.852734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42719 (local address 127.0.0.1:50804)
0927 02:43:13.852834 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:13.852880 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 02:43:13.853040 23120 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42719: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42719: connect: Connection refused (error 111)
W20260927 02:43:13.859050 23132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:13.859057 23133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:13.859064 23135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:43:13.859300 23127 server_base.cc:1034] running on GCE node
I20260927 02:43:13.859908 23127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:43:13.860918 23127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:43:13.862087 23127 hybrid_clock.cc:648] HybridClock initialized: now 1790476993862063 us; error 35 us; skew 500 ppm
Sep 27 02:43:13 dist-test-slave-zv8z krb5kdc[23039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:43:13.864645 23127 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 02:43:13.865501 23127 webserver.cc:460] Webserver started at http://127.0.98.61:42513/ using document root <none> and password file <none>
I20260927 02:43:13.866034 23127 fs_manager.cc:359] Metadata directory not provided
I20260927 02:43:13.866101 23127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:43:13.866314 23127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:43:13.868250 23127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/minicluster-data/master-1/data/instance:
uuid: "bdb807cc5f6242fc92d58d701661c820"
format_stamp: "Formatted at 2026-09-27 02:43:13 on dist-test-slave-zv8z"
server_key: "a720a6de7d7e0228758a2a66ca3fc5b6"
server_key_iv: "f1fd256d15534ec9a0ccc7463dcc731e"
server_key_version: "encryptionkey@0"
I20260927 02:43:13.868762 23127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/minicluster-data/master-1/wal/instance:
uuid: "bdb807cc5f6242fc92d58d701661c820"
format_stamp: "Formatted at 2026-09-27 02:43:13 on dist-test-slave-zv8z"
server_key: "a720a6de7d7e0228758a2a66ca3fc5b6"
server_key_iv: "f1fd256d15534ec9a0ccc7463dcc731e"
server_key_version: "encryptionkey@0"
I20260927 02:43:13.872287 23127 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260927 02:43:13.874598 23142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:13.875762 23127 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:43:13.875880 23127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/minicluster-data/master-1/wal
uuid: "bdb807cc5f6242fc92d58d701661c820"
format_stamp: "Formatted at 2026-09-27 02:43:13 on dist-test-slave-zv8z"
server_key: "a720a6de7d7e0228758a2a66ca3fc5b6"
server_key_iv: "f1fd256d15534ec9a0ccc7463dcc731e"
server_key_version: "encryptionkey@0"
I20260927 02:43:13.875965 23127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:43:13.892035 23127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:43:13.894578 23127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:43:13.894737 23127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:43:13.901058 23127 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:42719
I20260927 02:43:13.901368 23194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:42719 every 8 connection(s)
I20260927 02:43:13.901934 23127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/minicluster-data/master-1/data/info.pb
I20260927 02:43:13.904495 23195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:43:13.907924 23195 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36831 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:43:13.910449 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 23127
I20260927 02:43:13.910574 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/minicluster-data/master-1/wal/instance
I20260927 02:43:13.910861 392 external_mini_cluster.cc:1428] Setting key 8d0a8cf4575428025fa0004ce015ef9c
Sep 27 02:43:13 dist-test-slave-zv8z krb5kdc[23039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260927 02:43:13.920089 23123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:13.909609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54362 (local address 127.0.98.62:36831)
0927 02:43:13.909746 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:13.909749 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:13.910502 (+ 753us) server_negotiation.cc:407] Connection header received
0927 02:43:13.911462 (+ 960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:13.911466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:13.911526 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:13.911598 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:13.912696 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:13.913217 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:13.914339 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:13.914570 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:13.914662 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:13.916825 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:13.916828 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:13.916830 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:13.916853 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:13.918163 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:13.918645 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:13.918648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:13.918650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:13.918725 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:13.919022 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:13.919025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:13.919027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:13.919210 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:13.919292 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:13.919830 (+ 538us) server_negotiation.cc:299] Negotiation successful
0927 02:43:13.919937 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 02:43:13.921002 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:13.909919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36831 (local address 127.0.0.1:54362)
0927 02:43:13.910369 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:13.910410 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:13.911271 (+ 861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:13.911764 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:13.911770 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:13.912117 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:43:13.912551 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:13.912573 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:13.913329 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:13.913332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:13.914215 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:13.914224 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:13.914705 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:13.914709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:13.914860 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:13.915371 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:43:13.915386 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:43:13.916678 (+ 1292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:43:13.918286 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:13.918292 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:13.918305 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:13.918551 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:13.918802 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:13.918805 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:13.918808 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:13.918931 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:13.919303 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:43:13.919309 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:43:13.919554 (+ 245us) client_negotiation.cc:769] Sending connection context
0927 02:43:13.919739 (+ 185us) client_negotiation.cc:240] Negotiation successful
0927 02:43:13.920041 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":124,"threads_started":1}
I20260927 02:43:13.922212 23195 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42719 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:43:13 dist-test-slave-zv8z krb5kdc[23039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 02:43:13.932301 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:13.922686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42719 (local address 127.0.0.1:50812)
0927 02:43:13.922839 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:13.922853 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:13.922921 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:13.923374 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:13.923377 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:13.923532 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:43:13.923630 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:13.923637 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:13.924554 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:13.924558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:13.925388 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:13.925399 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:13.925949 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:13.925953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:13.926061 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:13.926487 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:43:13.926501 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:43:13.929264 (+ 2763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:43:13.930971 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:13.930973 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:13.930975 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:13.931169 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:13.931424 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:13.931426 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:13.931427 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:13.931498 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:13.931868 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:43:13.931870 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:43:13.931942 (+ 72us) client_negotiation.cc:769] Sending connection context
0927 02:43:13.932015 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 02:43:13.932148 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260927 02:43:13.932397 23200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:13.922769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50812 (local address 127.0.98.61:42719)
0927 02:43:13.923040 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:13.923050 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:13.923070 (+ 20us) server_negotiation.cc:407] Connection header received
0927 02:43:13.923109 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:13.923114 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:13.923252 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:13.923351 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:13.923749 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:13.924455 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:13.925492 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:13.925841 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:13.925916 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:13.929377 (+ 3461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:13.929385 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:13.929394 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:13.929414 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:13.930867 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:13.931288 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:13.931292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:13.931293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:13.931358 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:13.931577 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:13.931579 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:13.931581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:13.931784 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:13.931874 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:13.932074 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 02:43:13.932200 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":107,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 02:43:13.933509 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
/tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39335
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36831,127.0.98.61:42719,127.0.98.60:39335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:43:13.933972 23195 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39335 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:43:13.934631 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:13.934377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39335 (local address 127.0.0.1:49128)
0927 02:43:13.934486 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:13.934566 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 02:43:13.934827 23146 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39335: connect: Connection refused (error 111)
I20260927 02:43:13.935593 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:13.935419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39335 (local address 127.0.0.1:49130)
0927 02:43:13.935509 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:13.935548 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 02:43:13.935736 23195 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39335: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39335: connect: Connection refused (error 111)
I20260927 02:43:13.944187 23120 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42719 } attempt: 3
Sep 27 02:43:13 dist-test-slave-zv8z krb5kdc[23039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 02:43:13.952744 23200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:13.944665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50824 (local address 127.0.98.61:42719)
0927 02:43:13.944775 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:13.944778 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:13.944791 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:43:13.944931 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:13.944933 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:13.944972 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:13.945024 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:13.945454 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:13.945960 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:13.947090 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:13.947331 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:13.947417 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:13.949965 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:13.949968 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:13.949970 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:13.949991 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:13.951290 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:13.951762 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:13.951765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:13.951767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:13.951842 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:13.952079 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:13.952081 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:13.952083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:13.952252 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:13.952319 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:13.952554 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 02:43:13.952637 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 02:43:13.952800 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:13.944606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42719 (local address 127.0.0.1:50824)
0927 02:43:13.944735 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:13.944750 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:13.944839 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:13.945057 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:13.945059 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:13.945241 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:43:13.945348 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:13.945354 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:13.946057 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:13.946060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:13.946955 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:13.946965 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:13.947441 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:13.947444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:13.947551 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:13.947974 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:43:13.947988 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:43:13.949848 (+ 1860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:43:13.951388 (+ 1540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:13.951391 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:13.951394 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:13.951669 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:13.951917 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:13.951920 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:13.951922 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:13.952006 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:13.952326 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:43:13.952329 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:43:13.952418 (+ 89us) client_negotiation.cc:769] Sending connection context
0927 02:43:13.952524 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 02:43:13.952652 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 02:43:13.953752 23120 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39335 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:43:13.954373 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:13.954143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39335 (local address 127.0.0.1:49142)
0927 02:43:13.954244 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:13.954313 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 02:43:13.954970 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:13.954757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39335 (local address 127.0.0.1:49146)
0927 02:43:13.954873 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:13.954911 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260927 02:43:13.955101 23120 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39335: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39335: connect: Connection refused (error 111)
I20260927 02:43:13.984988 23195 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39335 } attempt: 1
I20260927 02:43:13.985817 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:13.985499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39335 (local address 127.0.0.1:49154)
0927 02:43:13.985656 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:13.985725 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 02:43:13.986282 23120 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39335 } attempt: 1
I20260927 02:43:13.986560 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:13.986304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39335 (local address 127.0.0.1:49164)
0927 02:43:13.986439 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:13.986489 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 02:43:13.986845 23195 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39335: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39335: connect: Connection refused (error 111)
I20260927 02:43:13.986927 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:13.986679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39335 (local address 127.0.0.1:49176)
0927 02:43:13.986806 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:13.986855 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 02:43:13.987547 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:13.987358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39335 (local address 127.0.0.1:49178)
0927 02:43:13.987454 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:13.987497 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 02:43:13.987665 23120 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39335: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39335: connect: Connection refused (error 111)
W20260927 02:43:14.040822 23201 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 02:43:14.041078 23201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:43:14.041101 23201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:43:14.041158 23201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:43:14.045164 23201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:43:14.045213 23201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:43:14.045231 23201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:43:14.045250 23201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:43:14.045264 23201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:43:14.050383 23201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36831,127.0.98.61:42719,127.0.98.60:39335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 01:57:39 UTC on ccycloud.quasar-jqbvls.root.comops.site
build id 4675614
I20260927 02:43:14.051326 23201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:43:14.052291 23201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:43:14.055016 23195 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39335 } attempt: 2
I20260927 02:43:14.055848 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:14.055532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39335 (local address 127.0.0.1:49190)
0927 02:43:14.055691 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:14.055760 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 02:43:14.056655 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:14.056383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39335 (local address 127.0.0.1:49196)
0927 02:43:14.056534 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:14.056587 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260927 02:43:14.056807 23195 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39335: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39335: connect: Connection refused (error 111)
I20260927 02:43:14.060544 23201 server_base.cc:1034] running on GCE node
W20260927 02:43:14.060525 23208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:14.060509 23207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:43:14.060786 23210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:43:14.061344 23201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:43:14.062492 23201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:43:14.062785 23120 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39335 } attempt: 2
I20260927 02:43:14.063400 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:14.063170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39335 (local address 127.0.0.1:49204)
0927 02:43:14.063286 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:14.063334 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 02:43:14.063647 23201 hybrid_clock.cc:648] HybridClock initialized: now 1790476994063627 us; error 28 us; skew 500 ppm
I20260927 02:43:14.064025 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:14.063833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39335 (local address 127.0.0.1:49216)
0927 02:43:14.063928 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:14.063969 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 02:43:14.064126 23120 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39335: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39335: connect: Connection refused (error 111)
Sep 27 02:43:14 dist-test-slave-zv8z krb5kdc[23039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476994, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:43:14.066223 23201 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 02:43:14.067041 23201 webserver.cc:460] Webserver started at http://127.0.98.60:37285/ using document root <none> and password file <none>
I20260927 02:43:14.067546 23201 fs_manager.cc:359] Metadata directory not provided
I20260927 02:43:14.067611 23201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:43:14.067790 23201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:43:14.069679 23201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/minicluster-data/master-2/data/instance:
uuid: "274059cb82b64e6b84e5517f5cfaab86"
format_stamp: "Formatted at 2026-09-27 02:43:14 on dist-test-slave-zv8z"
server_key: "016501840fc53f11167308b555792b67"
server_key_iv: "5bc5bbdd968f1fa893034fa01923ac6a"
server_key_version: "encryptionkey@0"
I20260927 02:43:14.070183 23201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/minicluster-data/master-2/wal/instance:
uuid: "274059cb82b64e6b84e5517f5cfaab86"
format_stamp: "Formatted at 2026-09-27 02:43:14 on dist-test-slave-zv8z"
server_key: "016501840fc53f11167308b555792b67"
server_key_iv: "5bc5bbdd968f1fa893034fa01923ac6a"
server_key_version: "encryptionkey@0"
I20260927 02:43:14.073607 23201 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 02:43:14.075909 23217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:43:14.077051 23201 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:43:14.077198 23201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/minicluster-data/master-2/wal
uuid: "274059cb82b64e6b84e5517f5cfaab86"
format_stamp: "Formatted at 2026-09-27 02:43:14 on dist-test-slave-zv8z"
server_key: "016501840fc53f11167308b555792b67"
server_key_iv: "5bc5bbdd968f1fa893034fa01923ac6a"
server_key_version: "encryptionkey@0"
I20260927 02:43:14.077280 23201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:43:14.092182 23201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:43:14.095130 23201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:43:14.095299 23201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:43:14.101476 23201 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39335
I20260927 02:43:14.101739 23269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39335 every 8 connection(s)
I20260927 02:43:14.102407 23201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/minicluster-data/master-2/data/info.pb
I20260927 02:43:14.105072 23270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:43:14.108567 23270 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36831 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:43:14.109040 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu as pid 23201
I20260927 02:43:14.109161 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBFiRWZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790476718023131-392-0/minicluster-data/master-2/wal/instance
I20260927 02:43:14.109406 392 external_mini_cluster.cc:1428] Setting key 2b4f2bae25ef153b3c59229f7f53014d
Sep 27 02:43:14 dist-test-slave-zv8z krb5kdc[23039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476994, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 02:43:14 dist-test-slave-zv8z krb5kdc[23039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 02:43:14.121133 23123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:14.110427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54370 (local address 127.0.98.62:36831)
0927 02:43:14.110553 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:14.110558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:14.111102 (+ 544us) server_negotiation.cc:407] Connection header received
0927 02:43:14.111923 (+ 821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:14.111926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:14.112065 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:14.112153 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:14.113013 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:14.113541 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:14.114613 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:14.114886 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:14.114978 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:14.117608 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:14.117613 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:14.117616 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:14.117645 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:14.119106 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:14.119651 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:14.119654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:14.119655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:14.119725 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:14.120043 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:14.120046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:14.120048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:14.120273 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:14.120366 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:14.120878 (+ 512us) server_negotiation.cc:299] Negotiation successful
0927 02:43:14.120972 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 02:43:14.121989 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:14.110548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49230 (local address 127.0.98.60:39335)
0927 02:43:14.110935 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:14.110952 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:14.110989 (+ 37us) server_negotiation.cc:407] Connection header received
0927 02:43:14.111705 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:14.111724 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:14.111919 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:14.112056 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:14.112780 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:14.113710 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:14.114682 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:14.115129 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:14.115230 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:14.118153 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:14.118158 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:14.118168 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:14.118187 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:14.119890 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:14.120375 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:14.120380 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:14.120382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:14.120459 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:14.120701 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:14.120704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:14.120706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:14.120946 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:14.121111 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:14.121392 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 02:43:14.121515 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":270,"spinlock_wait_cycles":54016,"thread_start_us":132,"threads_started":1}
I20260927 02:43:14.121999 23273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:14.110547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36831 (local address 127.0.0.1:54370)
0927 02:43:14.110976 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:14.111010 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:14.111718 (+ 708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:14.112215 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:14.112220 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:14.112565 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:43:14.112835 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:14.112849 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:14.113644 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:14.113647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:14.114478 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:14.114486 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:14.115038 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:14.115041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:14.115201 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:14.115955 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:43:14.115980 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:43:14.117434 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:43:14.119208 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:14.119217 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:14.119227 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:14.119563 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:14.119805 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:14.119808 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:14.119810 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:14.119966 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:14.120369 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:43:14.120375 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:43:14.120645 (+ 270us) client_negotiation.cc:769] Sending connection context
0927 02:43:14.120832 (+ 187us) client_negotiation.cc:240] Negotiation successful
0927 02:43:14.121129 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":45,"spinlock_wait_cycles":1280,"thread_start_us":108,"threads_started":1}
I20260927 02:43:14.123144 23270 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42719 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:43:14 dist-test-slave-zv8z krb5kdc[23039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476994, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 02:43:14 dist-test-slave-zv8z krb5kdc[23039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:43:14.130750 23273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:14.123579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42719 (local address 127.0.0.1:50836)
0927 02:43:14.123700 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:14.123713 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:14.123779 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:14.123986 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:14.123989 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:14.124154 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:43:14.124254 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:14.124261 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:14.125047 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:14.125049 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:14.125602 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:14.125611 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:14.125998 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:14.126001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:14.126104 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:14.126574 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:43:14.126585 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:43:14.127807 (+ 1222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:43:14.129416 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:14.129419 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:14.129421 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:14.129600 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:14.129839 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:14.129842 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:14.129844 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:14.129916 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:14.130267 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:43:14.130271 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:43:14.130354 (+ 83us) client_negotiation.cc:769] Sending connection context
0927 02:43:14.130478 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 02:43:14.130596 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 02:43:14.130757 23200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:14.123696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50836 (local address 127.0.98.61:42719)
0927 02:43:14.123810 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:14.123815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:14.123828 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:43:14.123866 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:14.123868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:14.123903 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:14.123965 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:14.124373 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:14.124943 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:14.125709 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:14.125910 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:14.125982 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:14.127948 (+ 1966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:14.127952 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:14.127955 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:14.127984 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:14.129319 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:14.129684 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:14.129687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:14.129689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:14.129759 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:14.129992 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:14.129995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:14.129997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:14.130165 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:14.130269 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:14.130546 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 02:43:14.130640 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 02:43:14.131601 23270 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39335 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:43:14.132223 23123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:14.124215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54374 (local address 127.0.98.62:36831)
0927 02:43:14.124319 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:14.124323 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:14.124336 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:43:14.124411 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:14.124413 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:14.124452 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:14.124522 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:14.124987 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:14.125506 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:14.126374 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:14.126580 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:14.126660 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:14.129047 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:14.129050 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:14.129052 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:14.129073 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:14.130349 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:14.130769 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:14.130773 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:14.130775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:14.131085 (+ 310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:14.131335 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:14.131338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:14.131339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:14.131525 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:14.131757 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:14.131831 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 02:43:14.131942 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Sep 27 02:43:14 dist-test-slave-zv8z krb5kdc[23039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476994, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 02:43:14 dist-test-slave-zv8z krb5kdc[23039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 02:43:14.139369 23273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:14.132065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39335 (local address 127.0.0.1:49246)
0927 02:43:14.132265 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:14.132277 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:14.132352 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:14.132559 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:14.132562 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:14.132709 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:43:14.132794 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:14.132799 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:14.133588 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:14.133591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:14.134072 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:14.134078 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:14.134559 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:14.134562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:14.134667 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:14.135045 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:43:14.135059 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:43:14.136202 (+ 1143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:43:14.137987 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:14.137991 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:14.137993 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:14.138223 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:14.138511 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:14.138514 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:14.138516 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:14.138608 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:14.138923 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:43:14.138926 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:43:14.139001 (+ 75us) client_negotiation.cc:769] Sending connection context
0927 02:43:14.139081 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 02:43:14.139198 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":16}
I20260927 02:43:14.139375 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:14.132181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49246 (local address 127.0.98.60:39335)
0927 02:43:14.132267 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:14.132270 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:14.132422 (+ 152us) server_negotiation.cc:407] Connection header received
0927 02:43:14.132444 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:14.132446 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:14.132477 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:14.132531 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:14.132879 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:14.133500 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:14.134164 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:14.134470 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:14.134552 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:14.136331 (+ 1779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:14.136334 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:14.136337 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:14.136359 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:14.137874 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:14.138331 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:14.138334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:14.138335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:14.138420 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:14.138688 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:14.138691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:14.138692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:14.138839 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:14.138918 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:14.139165 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 02:43:14.139247 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":26}
I20260927 02:43:14.141826 23200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:14.133299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50840 (local address 127.0.98.61:42719)
0927 02:43:14.133398 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:14.133401 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:14.133413 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:43:14.133512 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:14.133515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:14.133563 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:14.133627 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:14.134221 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:14.134720 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:14.135933 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:14.136247 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:14.136347 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:14.138875 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:14.138879 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:14.138881 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:14.138905 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:14.140413 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:14.140781 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:14.140784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:14.140786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:14.140858 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:14.141095 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:14.141098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:14.141100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:14.141256 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:14.141485 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:14.141585 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 02:43:14.141687 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 02:43:14.142678 23270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 274059cb82b64e6b84e5517f5cfaab86: Bootstrap starting.
I20260927 02:43:14.144987 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:43:14.145182 23270 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 274059cb82b64e6b84e5517f5cfaab86: Neither blocks nor log segments found. Creating new log.
I20260927 02:43:14.145928 23270 log.cc:824] T 00000000000000000000000000000000 P 274059cb82b64e6b84e5517f5cfaab86: Log is configured to *not* fsync() on all Append() calls
I20260927 02:43:14.146057 23094 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "936623e894d241ec8ce9fab364baddd8"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54374
I20260927 02:43:14.147857 23270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 274059cb82b64e6b84e5517f5cfaab86: No bootstrap required, opened a new log
I20260927 02:43:14.147962 23195 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39335 } attempt: 3
I20260927 02:43:14.151551 23270 raft_consensus.cc:348] T 00000000000000000000000000000000 P 274059cb82b64e6b84e5517f5cfaab86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936623e894d241ec8ce9fab364baddd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36831 } } peers { permanent_uuid: "bdb807cc5f6242fc92d58d701661c820" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42719 } } peers { permanent_uuid: "274059cb82b64e6b84e5517f5cfaab86" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39335 } }
I20260927 02:43:14.151790 23270 raft_consensus.cc:729] T 00000000000000000000000000000000 P 274059cb82b64e6b84e5517f5cfaab86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 274059cb82b64e6b84e5517f5cfaab86, State: Initialized, Role: FOLLOWER
I20260927 02:43:14.152143 23270 consensus_queue.cc:260] T 00000000000000000000000000000000 P 274059cb82b64e6b84e5517f5cfaab86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936623e894d241ec8ce9fab364baddd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36831 } } peers { permanent_uuid: "bdb807cc5f6242fc92d58d701661c820" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42719 } } peers { permanent_uuid: "274059cb82b64e6b84e5517f5cfaab86" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39335 } }
I20260927 02:43:14.152661 23276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 274059cb82b64e6b84e5517f5cfaab86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936623e894d241ec8ce9fab364baddd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36831 } } peers { permanent_uuid: "bdb807cc5f6242fc92d58d701661c820" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42719 } } peers { permanent_uuid: "274059cb82b64e6b84e5517f5cfaab86" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39335 } } }
I20260927 02:43:14.152793 23276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 274059cb82b64e6b84e5517f5cfaab86 [sys.catalog]: This master's current role is: FOLLOWER
Sep 27 02:43:14 dist-test-slave-zv8z krb5kdc[23039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 02:43:14.153502 23270 sys_catalog.cc:564] T 00000000000000000000000000000000 P 274059cb82b64e6b84e5517f5cfaab86 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:43:14.157091 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:14.148325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39335 (local address 127.0.0.1:49248)
0927 02:43:14.148482 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:14.148496 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:14.148566 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:14.148798 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:14.148801 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:14.148974 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:43:14.149069 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:14.149075 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:14.150089 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:14.150092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:14.151076 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:14.151088 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:14.151681 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:14.151685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:14.151766 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:14.152274 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:43:14.152291 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:43:14.153503 (+ 1212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:43:14.155343 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:14.155346 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:14.155348 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:14.155583 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:14.156069 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:14.156073 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:14.156075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:14.156163 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:14.156591 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:43:14.156594 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:43:14.156697 (+ 103us) client_negotiation.cc:769] Sending connection context
0927 02:43:14.156808 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 02:43:14.156941 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260927 02:43:14.157792 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:14.148402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49248 (local address 127.0.98.60:39335)
0927 02:43:14.148514 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:14.148518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:14.148532 (+ 14us) server_negotiation.cc:407] Connection header received
0927 02:43:14.148648 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:14.148651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:14.148693 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:14.148769 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:14.149195 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:14.149947 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:14.151200 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:14.151548 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:14.151667 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:14.153623 (+ 1956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:14.153626 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:14.153629 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:14.153659 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:14.155256 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:14.155685 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:14.155688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:14.155690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:14.155772 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:14.156237 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:14.156257 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:14.156259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:14.156488 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:14.156582 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:14.157553 (+ 971us) server_negotiation.cc:299] Negotiation successful
0927 02:43:14.157664 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260927 02:43:14.159255 23287 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 274059cb82b64e6b84e5517f5cfaab86: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:43:14.159373 23287 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:43:14.160462 23195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdb807cc5f6242fc92d58d701661c820: Bootstrap starting.
I20260927 02:43:14.157481 23201 master_runner.cc:186] Error getting master registration for 127.0.98.62:36831: OK, instance_id { permanent_uuid: "936623e894d241ec8ce9fab364baddd8" instance_seqno: 1790476993702904 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 02:43:14.160606 23201 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 02:43:14.162586 23195 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bdb807cc5f6242fc92d58d701661c820: Neither blocks nor log segments found. Creating new log.
I20260927 02:43:14.163246 23195 log.cc:824] T 00000000000000000000000000000000 P bdb807cc5f6242fc92d58d701661c820: Log is configured to *not* fsync() on all Append() calls
I20260927 02:43:14.163257 23120 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39335 } attempt: 3
I20260927 02:43:14.165704 23195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdb807cc5f6242fc92d58d701661c820: No bootstrap required, opened a new log
Sep 27 02:43:14 dist-test-slave-zv8z krb5kdc[23039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790476993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 02:43:14.169593 23195 raft_consensus.cc:348] T 00000000000000000000000000000000 P bdb807cc5f6242fc92d58d701661c820 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936623e894d241ec8ce9fab364baddd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36831 } } peers { permanent_uuid: "bdb807cc5f6242fc92d58d701661c820" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42719 } } peers { permanent_uuid: "274059cb82b64e6b84e5517f5cfaab86" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39335 } }
I20260927 02:43:14.169816 23195 raft_consensus.cc:729] T 00000000000000000000000000000000 P bdb807cc5f6242fc92d58d701661c820 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdb807cc5f6242fc92d58d701661c820, State: Initialized, Role: FOLLOWER
I20260927 02:43:14.170146 23195 consensus_queue.cc:260] T 00000000000000000000000000000000 P bdb807cc5f6242fc92d58d701661c820 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936623e894d241ec8ce9fab364baddd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36831 } } peers { permanent_uuid: "bdb807cc5f6242fc92d58d701661c820" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42719 } } peers { permanent_uuid: "274059cb82b64e6b84e5517f5cfaab86" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39335 } }
I20260927 02:43:14.170603 23290 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdb807cc5f6242fc92d58d701661c820 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936623e894d241ec8ce9fab364baddd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36831 } } peers { permanent_uuid: "bdb807cc5f6242fc92d58d701661c820" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42719 } } peers { permanent_uuid: "274059cb82b64e6b84e5517f5cfaab86" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39335 } } }
I20260927 02:43:14.170773 23290 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdb807cc5f6242fc92d58d701661c820 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:43:14.171360 23195 sys_catalog.cc:564] T 00000000000000000000000000000000 P bdb807cc5f6242fc92d58d701661c820 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:43:14.172775 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:14.163677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39335 (local address 127.0.0.1:49250)
0927 02:43:14.163793 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 02:43:14.163807 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:43:14.163879 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:43:14.164275 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:43:14.164278 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:43:14.164460 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:43:14.164563 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:14.164570 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:14.165562 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:14.165565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:14.166364 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:43:14.166375 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:43:14.167025 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:43:14.167029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:43:14.167116 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:14.167581 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:43:14.167596 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:43:14.169058 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:43:14.170928 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:14.170931 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:14.170933 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:14.171178 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:14.171546 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:43:14.171549 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:43:14.171551 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:43:14.171654 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:43:14.172266 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:43:14.172269 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:43:14.172376 (+ 107us) client_negotiation.cc:769] Sending connection context
0927 02:43:14.172475 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 02:43:14.172605 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 02:43:14.173429 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:14.163730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49250 (local address 127.0.98.60:39335)
0927 02:43:14.164015 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:14.164019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:14.164029 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:43:14.164059 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:14.164062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:14.164098 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:14.164166 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:14.164690 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:14.165452 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:14.166486 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:14.166816 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:14.166912 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:14.169204 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:14.169207 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:14.169210 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:14.169232 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:14.170798 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:14.171291 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:14.171295 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:14.171296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:14.171425 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:14.171763 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:14.171766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:14.171768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:14.172088 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:14.172917 (+ 829us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:14.173045 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 02:43:14.173218 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216}
I20260927 02:43:14.176533 23120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 936623e894d241ec8ce9fab364baddd8: Bootstrap starting.
W20260927 02:43:14.176771 23301 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bdb807cc5f6242fc92d58d701661c820: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:43:14.176882 23301 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:43:14.174499 23127 master_runner.cc:186] Error getting master registration for 127.0.98.62:36831: OK, instance_id { permanent_uuid: "936623e894d241ec8ce9fab364baddd8" instance_seqno: 1790476993702904 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 02:43:14.178076 23127 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 02:43:14.179141 23120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 936623e894d241ec8ce9fab364baddd8: Neither blocks nor log segments found. Creating new log.
I20260927 02:43:14.180004 23120 log.cc:824] T 00000000000000000000000000000000 P 936623e894d241ec8ce9fab364baddd8: Log is configured to *not* fsync() on all Append() calls
I20260927 02:43:14.182204 23120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 936623e894d241ec8ce9fab364baddd8: No bootstrap required, opened a new log
I20260927 02:43:14.185077 23120 raft_consensus.cc:348] T 00000000000000000000000000000000 P 936623e894d241ec8ce9fab364baddd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936623e894d241ec8ce9fab364baddd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36831 } } peers { permanent_uuid: "bdb807cc5f6242fc92d58d701661c820" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42719 } } peers { permanent_uuid: "274059cb82b64e6b84e5517f5cfaab86" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39335 } }
I20260927 02:43:14.185303 23120 raft_consensus.cc:729] T 00000000000000000000000000000000 P 936623e894d241ec8ce9fab364baddd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 936623e894d241ec8ce9fab364baddd8, State: Initialized, Role: FOLLOWER
I20260927 02:43:14.185648 23120 consensus_queue.cc:260] T 00000000000000000000000000000000 P 936623e894d241ec8ce9fab364baddd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936623e894d241ec8ce9fab364baddd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36831 } } peers { permanent_uuid: "bdb807cc5f6242fc92d58d701661c820" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42719 } } peers { permanent_uuid: "274059cb82b64e6b84e5517f5cfaab86" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39335 } }
I20260927 02:43:14.186187 23304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 936623e894d241ec8ce9fab364baddd8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936623e894d241ec8ce9fab364baddd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36831 } } peers { permanent_uuid: "bdb807cc5f6242fc92d58d701661c820" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42719 } } peers { permanent_uuid: "274059cb82b64e6b84e5517f5cfaab86" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39335 } } }
I20260927 02:43:14.186470 23304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 936623e894d241ec8ce9fab364baddd8 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:43:14.186947 23120 sys_catalog.cc:564] T 00000000000000000000000000000000 P 936623e894d241ec8ce9fab364baddd8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:43:14.187023 23094 raft_consensus.cc:482] T 00000000000000000000000000000000 P 936623e894d241ec8ce9fab364baddd8 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 02:43:14.187134 23094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 936623e894d241ec8ce9fab364baddd8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:43:14.188338 23094 raft_consensus.cc:504] T 00000000000000000000000000000000 P 936623e894d241ec8ce9fab364baddd8 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936623e894d241ec8ce9fab364baddd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36831 } } peers { permanent_uuid: "bdb807cc5f6242fc92d58d701661c820" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42719 } } peers { permanent_uuid: "274059cb82b64e6b84e5517f5cfaab86" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39335 } }
I20260927 02:43:14.189241 23094 leader_election.cc:288] T 00000000000000000000000000000000 P 936623e894d241ec8ce9fab364baddd8 [CANDIDATE]: Term 1 election: Requested vote from peers bdb807cc5f6242fc92d58d701661c820 (127.0.98.61:42719), 274059cb82b64e6b84e5517f5cfaab86 (127.0.98.60:39335)
I20260927 02:43:14.190343 23169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "936623e894d241ec8ce9fab364baddd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bdb807cc5f6242fc92d58d701661c820"
I20260927 02:43:14.190343 23244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "936623e894d241ec8ce9fab364baddd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "274059cb82b64e6b84e5517f5cfaab86"
I20260927 02:43:14.190567 23244 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 274059cb82b64e6b84e5517f5cfaab86 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:43:14.190567 23169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bdb807cc5f6242fc92d58d701661c820 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:43:14.191756 23169 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bdb807cc5f6242fc92d58d701661c820 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 936623e894d241ec8ce9fab364baddd8 in term 1.
I20260927 02:43:14.191757 23244 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 274059cb82b64e6b84e5517f5cfaab86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 936623e894d241ec8ce9fab364baddd8 in term 1.
I20260927 02:43:14.192137 23071 leader_election.cc:302] T 00000000000000000000000000000000 P 936623e894d241ec8ce9fab364baddd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 274059cb82b64e6b84e5517f5cfaab86, 936623e894d241ec8ce9fab364baddd8; no voters:
I20260927 02:43:14.192377 23304 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 936623e894d241ec8ce9fab364baddd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:43:14.192624 23304 raft_consensus.cc:686] T 00000000000000000000000000000000 P 936623e894d241ec8ce9fab364baddd8 [term 1 LEADER]: Becoming Leader. State: Replica: 936623e894d241ec8ce9fab364baddd8, State: Running, Role: LEADER
I20260927 02:43:14.192986 23304 consensus_queue.cc:237] T 00000000000000000000000000000000 P 936623e894d241ec8ce9fab364baddd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936623e894d241ec8ce9fab364baddd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36831 } } peers { permanent_uuid: "bdb807cc5f6242fc92d58d701661c820" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42719 } } peers { permanent_uuid: "274059cb82b64e6b84e5517f5cfaab86" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39335 } }
I20260927 02:43:14.195372 23304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 936623e894d241ec8ce9fab364baddd8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 936623e894d241ec8ce9fab364baddd8. Latest consensus state: current_term: 1 leader_uuid: "936623e894d241ec8ce9fab364baddd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936623e894d241ec8ce9fab364baddd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36831 } } peers { permanent_uuid: "bdb807cc5f6242fc92d58d701661c820" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42719 } } peers { permanent_uuid: "274059cb82b64e6b84e5517f5cfaab86" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39335 } } }
I20260927 02:43:14.195612 23304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 936623e894d241ec8ce9fab364baddd8 [sys.catalog]: This master's current role is: LEADER
I20260927 02:43:14.196203 23324 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:43:14.196444 23052 master_runner.cc:263] No leader master found from master 936623e894d241ec8ce9fab364baddd8
I20260927 02:43:14.196499 23052 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 02:43:14.199232 23324 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260927 02:43:14.199713 23073 tablet.cc:2369] T 00000000000000000000000000000000 P 936623e894d241ec8ce9fab364baddd8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:43:14.210491 23200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:14.194593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50848 (local address 127.0.98.61:42719)
0927 02:43:14.194696 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:14.194700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:14.195778 (+ 1078us) server_negotiation.cc:407] Connection header received
0927 02:43:14.195964 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:14.195967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:14.196015 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:14.196065 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:14.196996 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:14.197770 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:14.198954 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:14.199279 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:14.199345 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:14.203096 (+ 3751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:14.203099 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:14.203102 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:14.203127 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:14.204927 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:14.207147 (+ 2220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:14.207151 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:14.207153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:14.207235 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:14.207509 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:14.207512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:14.207514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:14.207723 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:14.207828 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:14.210196 (+ 2368us) server_negotiation.cc:299] Negotiation successful
0927 02:43:14.210330 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 02:43:14.214044 23244 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 274059cb82b64e6b84e5517f5cfaab86 [term 1 FOLLOWER]: Refusing update from remote peer 936623e894d241ec8ce9fab364baddd8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:43:14.214792 23320 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 936623e894d241ec8ce9fab364baddd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "274059cb82b64e6b84e5517f5cfaab86" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:43:14.215183 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:14.195242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49262 (local address 127.0.98.60:39335)
0927 02:43:14.195343 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:14.195347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:14.201070 (+ 5723us) server_negotiation.cc:407] Connection header received
0927 02:43:14.201250 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:14.201253 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:14.201298 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:14.201345 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:14.202086 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:14.202858 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:14.203948 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:14.204285 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:14.204358 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:14.208132 (+ 3774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:14.208135 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:14.208138 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:14.208162 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:14.209698 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:14.212855 (+ 3157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:14.212858 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:14.212861 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:14.212944 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:14.213276 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:14.213280 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:14.213281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:14.213488 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:14.213589 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:14.214908 (+ 1319us) server_negotiation.cc:299] Negotiation successful
0927 02:43:14.215035 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 02:43:14.218680 23276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 274059cb82b64e6b84e5517f5cfaab86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 936623e894d241ec8ce9fab364baddd8. Latest consensus state: current_term: 1 leader_uuid: "936623e894d241ec8ce9fab364baddd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936623e894d241ec8ce9fab364baddd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36831 } } peers { permanent_uuid: "bdb807cc5f6242fc92d58d701661c820" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42719 } } peers { permanent_uuid: "274059cb82b64e6b84e5517f5cfaab86" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39335 } } }
I20260927 02:43:14.218786 23276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 274059cb82b64e6b84e5517f5cfaab86 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:43:14.219794 23123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:14.196183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54380 (local address 127.0.98.62:36831)
0927 02:43:14.196375 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:14.196379 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:14.201238 (+ 4859us) server_negotiation.cc:407] Connection header received
0927 02:43:14.201379 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:14.201382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:14.201424 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:14.201468 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:14.202167 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:14.202897 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:14.211893 (+ 8996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:14.212232 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:14.212302 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:14.216461 (+ 4159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:43:14.216465 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:43:14.216468 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:43:14.216493 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:43:14.218036 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:14.218522 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:14.218525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:14.218527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:14.218604 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:43:14.218878 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:43:14.218881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:43:14.218883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:43:14.219109 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:43:14.219211 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:14.219523 (+ 312us) server_negotiation.cc:299] Negotiation successful
0927 02:43:14.219638 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260927 02:43:14.219774 23169 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bdb807cc5f6242fc92d58d701661c820 [term 1 FOLLOWER]: Refusing update from remote peer 936623e894d241ec8ce9fab364baddd8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:43:14.220477 23320 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 936623e894d241ec8ce9fab364baddd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdb807cc5f6242fc92d58d701661c820" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:43:14.224082 23290 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdb807cc5f6242fc92d58d701661c820 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 936623e894d241ec8ce9fab364baddd8. Latest consensus state: current_term: 1 leader_uuid: "936623e894d241ec8ce9fab364baddd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936623e894d241ec8ce9fab364baddd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36831 } } peers { permanent_uuid: "bdb807cc5f6242fc92d58d701661c820" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42719 } } peers { permanent_uuid: "274059cb82b64e6b84e5517f5cfaab86" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39335 } } }
I20260927 02:43:14.224191 23290 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdb807cc5f6242fc92d58d701661c820 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:43:14.228611 23320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 936623e894d241ec8ce9fab364baddd8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "936623e894d241ec8ce9fab364baddd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936623e894d241ec8ce9fab364baddd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36831 } } peers { permanent_uuid: "bdb807cc5f6242fc92d58d701661c820" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42719 } } peers { permanent_uuid: "274059cb82b64e6b84e5517f5cfaab86" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39335 } } }
I20260927 02:43:14.228734 23320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 936623e894d241ec8ce9fab364baddd8 [sys.catalog]: This master's current role is: LEADER
I20260927 02:43:14.229912 23276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 274059cb82b64e6b84e5517f5cfaab86 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "936623e894d241ec8ce9fab364baddd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936623e894d241ec8ce9fab364baddd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36831 } } peers { permanent_uuid: "bdb807cc5f6242fc92d58d701661c820" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42719 } } peers { permanent_uuid: "274059cb82b64e6b84e5517f5cfaab86" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39335 } } }
I20260927 02:43:14.230011 23276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 274059cb82b64e6b84e5517f5cfaab86 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:43:14.230432 23327 mvcc.cc:205] Tried to move back new op lower bound from 7333793768293261312 to 7333793768220659712. Current Snapshot: MvccSnapshot[applied={T|T < 7333793768293261312}]
I20260927 02:43:14.235981 23304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 936623e894d241ec8ce9fab364baddd8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "936623e894d241ec8ce9fab364baddd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936623e894d241ec8ce9fab364baddd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36831 } } peers { permanent_uuid: "bdb807cc5f6242fc92d58d701661c820" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42719 } } peers { permanent_uuid: "274059cb82b64e6b84e5517f5cfaab86" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39335 } } }
I20260927 02:43:14.236086 23304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 936623e894d241ec8ce9fab364baddd8 [sys.catalog]: This master's current role is: LEADER
I20260927 02:43:14.237124 23290 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdb807cc5f6242fc92d58d701661c820 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "936623e894d241ec8ce9fab364baddd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936623e894d241ec8ce9fab364baddd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36831 } } peers { permanent_uuid: "bdb807cc5f6242fc92d58d701661c820" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42719 } } peers { permanent_uuid: "274059cb82b64e6b84e5517f5cfaab86" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39335 } } }
I20260927 02:43:14.237226 23290 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdb807cc5f6242fc92d58d701661c820 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:43:14.237891 23330 mvcc.cc:205] Tried to move back new op lower bound from 7333793768293261312 to 7333793768220659712. Current Snapshot: MvccSnapshot[applied={T|T < 7333793768293261312}]
I20260927 02:43:14.239924 23324 catalog_manager.cc:1347] Generated new cluster ID: a24ab6d7f1464231b986a03b3f9de109
I20260927 02:43:14.239981 23324 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:43:14.275897 23324 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:43:14.277211 23324 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:43:14.310626 23324 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 936623e894d241ec8ce9fab364baddd8: Generated new TSK 0
I20260927 02:43:14.311460 23324 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 02:43:14.335361 23123 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:43:14.332153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54388 (local address 127.0.98.62:36831)
0927 02:43:14.332282 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:14.332287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:14.334245 (+ 1958us) server_negotiation.cc:407] Connection header received
0927 02:43:14.334479 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:14.334482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:14.334543 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:14.334596 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:14.335227 (+ 631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54388: BlockingRecv error: recv got EOF from 127.0.0.1:54388 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260927 02:43:14.336685 23200 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:43:14.332664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50856 (local address 127.0.98.61:42719)
0927 02:43:14.332769 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:14.332773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:14.335833 (+ 3060us) server_negotiation.cc:407] Connection header received
0927 02:43:14.335996 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:14.335999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:14.336048 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:14.336098 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:14.336565 (+ 467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50856: BlockingRecv error: recv got EOF from 127.0.0.1:50856 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260927 02:43:14.337970 23274 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:43:14.333666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49272 (local address 127.0.98.60:39335)
0927 02:43:14.333777 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:14.333781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:14.337129 (+ 3348us) server_negotiation.cc:407] Connection header received
0927 02:43:14.337290 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:14.337293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:14.337342 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:14.337389 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:14.337853 (+ 464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49272: BlockingRecv error: recv got EOF from 127.0.0.1:49272 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 02:43:14.342196 23200 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:43:14.340730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50868 (local address 127.0.98.61:42719)
0927 02:43:14.340831 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:14.340834 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:14.341294 (+ 460us) server_negotiation.cc:407] Connection header received
0927 02:43:14.341491 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:14.341494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:14.341579 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:14.341628 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:14.342093 (+ 465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50868: BlockingRecv error: recv got EOF from 127.0.0.1:50868 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 02:43:14.347127 23200 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:43:14.345639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50884 (local address 127.0.98.61:42719)
0927 02:43:14.345736 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:14.345740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:14.346220 (+ 480us) server_negotiation.cc:407] Connection header received
0927 02:43:14.346459 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:14.346462 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:14.346510 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:14.346575 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:14.347026 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50884: BlockingRecv error: recv got EOF from 127.0.0.1:50884 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260927 02:43:14.352779 23200 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:43:14.351149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50890 (local address 127.0.98.61:42719)
0927 02:43:14.351259 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:14.351263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:14.351829 (+ 566us) server_negotiation.cc:407] Connection header received
0927 02:43:14.352023 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:14.352027 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:14.352077 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:14.352128 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:14.352669 (+ 541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50890: BlockingRecv error: recv got EOF from 127.0.0.1:50890 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260927 02:43:14.360625 23200 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:43:14.358915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50904 (local address 127.0.98.61:42719)
0927 02:43:14.359055 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:14.359059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:14.359656 (+ 597us) server_negotiation.cc:407] Connection header received
0927 02:43:14.359862 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:14.359866 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:14.359917 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:14.359971 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:14.360500 (+ 529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50904: BlockingRecv error: recv got EOF from 127.0.0.1:50904 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260927 02:43:14.372143 23200 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:43:14.370912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50916 (local address 127.0.98.61:42719)
0927 02:43:14.371055 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:14.371059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:14.371259 (+ 200us) server_negotiation.cc:407] Connection header received
0927 02:43:14.371399 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:14.371402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:14.371454 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:14.371555 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:14.371997 (+ 442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50916: BlockingRecv error: recv got EOF from 127.0.0.1:50916 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260927 02:43:14.391518 23200 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:43:14.390324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50926 (local address 127.0.98.61:42719)
0927 02:43:14.390480 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:14.390485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:14.390633 (+ 148us) server_negotiation.cc:407] Connection header received
0927 02:43:14.390781 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:14.390783 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:14.390829 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:14.390898 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:14.391389 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50926: BlockingRecv error: recv got EOF from 127.0.0.1:50926 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260927 02:43:14.426689 23200 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:43:14.425504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50942 (local address 127.0.98.61:42719)
0927 02:43:14.425661 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:14.425665 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:14.425778 (+ 113us) server_negotiation.cc:407] Connection header received
0927 02:43:14.425927 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:14.425929 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:14.425976 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:14.426045 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:14.426559 (+ 514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50942: BlockingRecv error: recv got EOF from 127.0.0.1:50942 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260927 02:43:14.493607 23200 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:43:14.492541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50946 (local address 127.0.98.61:42719)
0927 02:43:14.492707 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:14.492712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:14.492841 (+ 129us) server_negotiation.cc:407] Connection header received
0927 02:43:14.492950 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:14.492952 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:14.493000 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:14.493072 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:14.493461 (+ 389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50946: BlockingRecv error: recv got EOF from 127.0.0.1:50946 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260927 02:43:14.624882 23200 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:43:14.623684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50962 (local address 127.0.98.61:42719)
0927 02:43:14.623881 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:14.623886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:14.624019 (+ 133us) server_negotiation.cc:407] Connection header received
0927 02:43:14.624176 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:14.624179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:14.624225 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:14.624294 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:14.624765 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50962: BlockingRecv error: recv got EOF from 127.0.0.1:50962 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260927 02:43:14.884095 23200 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:43:14.882947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50974 (local address 127.0.98.61:42719)
0927 02:43:14.883073 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:14.883078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:14.883263 (+ 185us) server_negotiation.cc:407] Connection header received
0927 02:43:14.883409 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:14.883411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:14.883458 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:14.883524 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:43:14.883981 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50974: BlockingRecv error: recv got EOF from 127.0.0.1:50974 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 02:43:15.160621 23287 catalog_manager.cc:1259] Loaded cluster ID: a24ab6d7f1464231b986a03b3f9de109
I20260927 02:43:15.160704 23287 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 274059cb82b64e6b84e5517f5cfaab86: loading cluster ID for follower catalog manager: success
I20260927 02:43:15.162495 23287 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 274059cb82b64e6b84e5517f5cfaab86: acquiring CA information for follower catalog manager: success
I20260927 02:43:15.163668 23287 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 274059cb82b64e6b84e5517f5cfaab86: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 02:43:15.178134 23301 catalog_manager.cc:1259] Loaded cluster ID: a24ab6d7f1464231b986a03b3f9de109
I20260927 02:43:15.178205 23301 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bdb807cc5f6242fc92d58d701661c820: loading cluster ID for follower catalog manager: success
I20260927 02:43:15.179976 23301 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bdb807cc5f6242fc92d58d701661c820: acquiring CA information for follower catalog manager: success
I20260927 02:43:15.181126 23301 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bdb807cc5f6242fc92d58d701661c820: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 02:43:15.401938 23399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:15.398184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50982 (local address 127.0.98.61:42719)
0927 02:43:15.398416 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:15.398420 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:15.398566 (+ 146us) server_negotiation.cc:407] Connection header received
0927 02:43:15.398637 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:15.398640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:15.398761 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:15.398865 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:43:15.399125 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:15.399671 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:15.400576 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:15.400804 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:15.400889 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:15.401116 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:43:15.401561 (+ 445us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:43:15.401650 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:15.401728 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 02:43:15.401777 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":93,"threads_started":1}
I20260927 02:43:15.408195 23405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:15.404892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49276 (local address 127.0.98.60:39335)
0927 02:43:15.405098 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:15.405103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:15.405118 (+ 15us) server_negotiation.cc:407] Connection header received
0927 02:43:15.405199 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:15.405201 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:15.405308 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:15.405407 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:43:15.405558 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:15.406072 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:15.406872 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:15.407094 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:15.407187 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:15.407450 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:43:15.407864 (+ 414us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:43:15.407951 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:15.408000 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 02:43:15.408046 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":57,"threads_started":1}
I20260927 02:43:15.415073 23399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:15.410886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50990 (local address 127.0.98.61:42719)
0927 02:43:15.411008 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:15.411013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:15.411424 (+ 411us) server_negotiation.cc:407] Connection header received
0927 02:43:15.411596 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:15.411599 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:15.411649 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:15.411693 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:43:15.412145 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:15.412936 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:15.413952 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:15.414318 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:15.414406 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:15.414645 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:43:15.414721 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:43:15.414784 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:15.414867 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 02:43:15.414919 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 02:43:15.415606 23412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:15.410841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54396 (local address 127.0.98.62:36831)
0927 02:43:15.411519 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:15.411523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:15.411536 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:43:15.411633 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:15.411636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:15.411788 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:15.411899 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:43:15.412182 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:15.412706 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:15.413824 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:15.414141 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:15.414240 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:15.414589 (+ 349us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:43:15.414933 (+ 344us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:43:15.415031 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:15.415396 (+ 365us) server_negotiation.cc:299] Negotiation successful
0927 02:43:15.415456 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":65,"threads_started":1}
I20260927 02:43:15.415935 23405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:43:15.411411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49282 (local address 127.0.98.60:39335)
0927 02:43:15.411546 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 02:43:15.411550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:43:15.411563 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:43:15.411670 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:43:15.411674 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:43:15.411713 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:43:15.411771 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:43:15.412181 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:15.412927 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:15.414745 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:43:15.415064 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:43:15.415160 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:43:15.415383 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:43:15.415462 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:43:15.415528 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 02:43:15.415666 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 02:43:15.415824 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 02:43:15.418702 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 23052
I20260927 02:43:15.423921 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 23127
I20260927 02:43:15.428928 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBFiRWZ/build/debug/bin/kudu with pid 23201
2026-09-27T02:43:15Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2972 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5921 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (277398 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST